Dedicated power meter to measure background light level in compressive imaging system

ABSTRACT

An imaging system and method that captures compressive sensing (CS) measurements of a received light stream, and also obtains samples of background light level (BGLL). The BGLL samples may be used to compensate the CS measurements for variations in the BGLL. The system includes: a light modulator to spatially modulate the received light stream with spatial patterns, and a lens to concentrate the modulated light stream onto a light detector. The samples of BGLL may be obtained in various ways: (a) injecting calibration patterns among the spatial patterns; (b) measuring complementary light reflected by digital micromirrors onto a secondary output path; (c) separating and measuring a portion of light from the optical input path; (d) low-pass filtering the CS measurements; and (e) employing a light power meter with its own separate input path. Also, the CS measurements may be high-pass filtered to attenuate background light variation.

PRIORITY CLAIM DATA

This application is a continuation of U.S. patent application Ser. No.13/193,556, filed Jul. 28, 2011, entitled “Low-Pass Filtering ofCompressive Imaging Measurements to Infer Light Level Variation”,invented by Baraniuk, Kelly, Bridge, Chatterjee and McMackin, whichclaims the benefit of priority to U.S. Provisional Application No.61/372,826, filed Aug. 11, 2010, entitled “Compressive Sensing Systemsand Methods”, invented by Baraniuk, Woods, Kelly, Bridge, Chatterjee andMcMackin. U.S. patent application Ser. No. 13/193,556 and U.S.Provisional Application No. 61/372,826 are hereby incorporated byreference in theirs entireties as though fully and completely set forthherein.

FIELD OF THE INVENTION

The present invention relates to the field of compressive sensing, andmore particularly, to a compressive imaging system and method thatmeasures light level using a power meter.

DESCRIPTION OF THE RELATED ART

According to Nyquist theory, a signal x(t) whose signal energy issupported on the frequency interval [−B,B] may be reconstructed fromsamples {x(nT)} of the signal x(t), provided the rate f_(S)=1/T_(S) atwhich the samples are captured is sufficiently high, i.e., provided thatf_(S) is greater than 2B. Similarly, for a signal whose signal energy issupported on the frequency interval [A,B], the signal may bereconstructed from samples captured with sample rate greater than B-A. Afundamental problem with any attempt to capture a signal x(t) accordingto Nyquist theory is the large number of samples that are generated,especially when B (or B-A) is large. The large number of samples istaxing on memory resources and on the capacity of transmission channels.

Nyquist theory is not limited to functions of time. Indeed, Nyquisttheory applies more generally to any function of one or more realvariables. For example, Nyquist theory applies to functions of twospatial variables such as images, to functions of time and two spatialvariables such as video, and to the functions used in multispectralimaging, hyperspectral imaging, medical imaging and a wide variety ofother applications. In the case of an image I(x,y) that depends onspatial variables x and y, the image may be reconstructed from samplesof the image, provided the samples are captured with sufficiently highspatial density. For example, given samples {I(nΔx,mΔy)} captured alonga rectangular grid, the horizontal and vertical densities 1/Δx and 1/Δyshould be respectively greater than 2B_(x) and 2B_(y), where B_(X) andB_(y) are the highest x and y spatial frequencies occurring in the imageI(x,y). The same problem of overwhelming data volume is experienced whenattempting to capture an image according to Nyquist theory. The modemtheory of compressive sensing is directed to such problems.

Compressive sensing relies on the observation that many signals (e.g.,images or video sequences) of practical interest are not onlyband-limited but also sparse or approximately sparse when representedusing an appropriate choice of transformation, for example, atransformation such as a Fourier transform, a wavelet transform or adiscrete cosine transform (DCT). A signal vector v is said to beK-sparse with respect to a given transformation T when thetransformation of the signal vector, Tv, has no more than K non-zerocoefficients. A signal vector v is said to be sparse with respect to agiven transformation T when it is K-sparse with respect to thattransformation for some integer K much smaller than the number L ofcomponents in the transformation vector Tv.

A signal vector v is said to be approximately K-sparse with respect to agiven transformation T when the coefficients of the transformationvector, Tv, are dominated by the K largest coefficients (i.e., largestin the sense of magnitude or absolute value). In other words, if the Klargest coefficients account for a high percentage of the energy in theentire set of coefficients, then the signal vector v is approximatelyK-sparse with respect to transformation T. A signal vector v is said tobe approximately sparse with respect to a given transformation T when itis approximately K-sparse with respect to the transformation T for someinteger K much less than the number L of components in thetransformation vector Tv.

Given a sensing device that captures images with N samples per image andin conformity to the Nyquist condition on spatial rates, it is often thecase that there exists some transformation and some integer K very muchsmaller than N such that the transform of each captured image will beapproximately K sparse. The set of K dominant coefficients may vary fromone image to the next. Furthermore, the value of K and the selection ofthe transformation may vary from one context (e.g., imaging application)to the next. Examples of typical transforms that might work in differentcontexts are the Fourier transform, the wavelet transform, the DCT, theGabor transform, etc.

Compressive sensing specifies a way of operating on the N samples of animage so as to generate a much smaller set of samples from which the Nsamples may be reconstructed, given knowledge of the transform underwhich the image is sparse (or approximately sparse). In particular,compressive sensing invites one to think of the N samples as a vector vin an N-dimensional space and to imagine projecting the vector v ontoeach vector in a series of M vectors {R(i)} in the N-dimensional space,where M is larger than K but still much smaller than N. Each projectiongives a corresponding real number s(i), e.g., according to theexpressions(0=<v,R(i)>,where the notation <v,R(i)> represents the inner product (or dotproduct) of the vector v and the vector R(i). Thus, the series of Mprojections gives a vector U including M real numbers. Compressivesensing theory further prescribes methods for reconstructing (orestimating) the vector v of N samples from the vector U of M realnumbers. For example, according to one method, one should determine thevector x that has the smallest length (in the sense of the L₁ norm)subject to the condition that ΦTx=U, where Φ is a matrix whose rows arethe transposes of the vectors R(i), where T is the transformation underwhich the image is K sparse or approximately K sparse.

Compressive sensing is important because, among other reasons, it allowsreconstruction of an image based on M measurements instead of the muchlarger number N of measurements recommended by Nyquist theory. Thus, forexample, a compressive sensing camera would be able to capture asignificantly larger number of images for a given size of image store,and/or, transmit a significantly larger number of images per unit timethrough a communication channel of given capacity.

As mentioned above, compressive sensing operates by projecting the imagevector v onto a series of M vectors. As discussed in U.S. patentapplication Ser. No. 11/379,688 (published as 2006/0239336 and inventedby Baraniuk et al.) and illustrated in FIG. 1, an imaging device (e.g.,camera) may be configured to take advantage of the compressive sensingparadigm by using a digital micromirror device (DMD) 40. An incidentlightfield 10 passes through a lens 20 and then interacts with the DMD40. The DMD includes a two-dimensional array of micromirrors, each ofwhich is configured to independently and controllably switch between twoorientation states. Each micromirror reflects a corresponding portion ofthe incident light field based on its instantaneous orientation. Anymicromirrors in a first of the two orientation states will reflect theircorresponding light portions so that they pass through lens 50. Anymicromirrors in a second of the two orientation states will reflecttheir corresponding light portions away from lens 50. Lens 50 serves toconcentrate the light portions from the micromirrors in the firstorientation state onto a photodiode (or photodetector) situated atlocation 60. Thus, the photodiode generates a signal whose amplitude atany given time represents a sum of the intensities of the light portionsfrom the micromirrors in the first orientation state.

The compressive sensing is implemented by driving the orientations ofthe micromirrors through a series of spatial patterns. Each spatialpattern specifies an orientation state for each of the micromirrors. Theoutput signal of the photodiode is digitized by an A/D converter 70. Inthis fashion, the imaging device is able to capture a series ofmeasurements {s(i)} that represent inner products (dot products) betweenthe incident light field and the series of spatial patterns withoutfirst acquiring the incident light field as a pixelized digital image.The incident light field corresponds to the vector v of the discussionabove, and the spatial patterns correspond to the vectors R(i) of thediscussion above.

The incident light field may be modeled by a function I(x,y,t) of twospatial variables and time. Assuming for the discussion of thisparagraph that the DMD comprises a rectangular array, the DMD implementsa spatial modulation of the incident light field so that the light fieldleaving the DMD in the direction of the lens 50 might be modeled by{I(nΔx,mΔy,t)*M(n,m,t)}where m and n are integer indices, where I(nΔx,mΔy,t) represents theportion of the light field that is incident upon that (n,m)^(th) mirrorof the DMD at time t. The function M(n,m,t) represents the orientationof the (n,m)^(th) mirror of the DMD at time t. At sampling times, thefunction M(n,m,t) equals one or zero, depending on the state of thedigital control signal that controls the (n,m)^(th) mirror. Thecondition M(n,m,t)=1 corresponds to the orientation state that reflectsonto the path that leads to the lens 50. The condition M(n,m,t)=0corresponds to the orientation state that reflects away from the lens50.

The lens 50 concentrates the spatially-modulated light field{I(nΔx,mΔy,t)*M(n,m,t)}onto a light sensitive surface of the photodiode. Thus, the lens and thephotodiode together implement a spatial summation of the light portionsin the spatially-modulated light field:

${S(t)} = {\sum\limits_{n,m}{{I\left( {{n\;\Delta\; x},{m\;\Delta\; y},t} \right)}{{M\left( {n,m,t} \right)}.}}}$Signal S(t) may be interpreted as the intensity at time t of theconcentrated spot of light impinging upon the light sensing surface ofthe photodiode. The A/D converter captures measurements of S(t). In thisfashion, the compressive sensing camera optically computes an innerproduct of the incident light field with each spatial pattern imposed onthe mirrors. The multiplication portion of the inner product isimplemented by the mirrors of the DMD. The summation portion of theinner product is implemented by the concentrating action of the lens andalso the integrating action of the photodiode.

One fundamental challenge with the imaging device of FIG. 1 is that thebackground light level (i.e., ambient illumination level) may varyduring the time interval over which the series of measurements {s(i)} isbeing captured. If one attempts to reconstruct the image based on themeasurements {s(i)} under the assumption that the background light levelwas constant, then the reconstruction may give less than satisfactoryresults.

SUMMARY

Embodiments of a system may be configured to operate on a received lightstream in order to capture compressive imaging measurements that areuseable to construct an image or video sequence. (The imagingmeasurements are referred to as being “compressive” because the numberof such measurements are much smaller than the number of pixels in theimage or video sequence that can be constructed from the imagingmeasurements.) The system also provides measurements of background lightlevel as function of time. The background light level measurements maybe used to compensate the compressive imaging measurements forvariations in the background light level, e.g., in order to approximatethe imaging measurements that would have been obtained if the backgroundlight level had remained constant during the measurement process.

The system may include a light modulation unit and a light sensingdevice. The light modulation unit is configured to modulate the receivedlight stream with a series of spatial patterns in order to produce amodulated light stream. The light modulation unit includes a pluralityof light modulating elements configured to modulate correspondingportions of the received light stream. Each of the spatial patternsspecifies an amount of modulation (e.g., a scalar multiplier) for eachof the light modulating elements.

The light modulating elements may be mirrors (e.g., an array ofmicromirrors). Each of the mirrors may be configured to controllablyswitch between two orientation states. Furthermore, each of the mirrorsmay be configured to (a) reflect a corresponding portion of the receivedlight stream onto a sensing path when the mirror is in a first of thetwo orientation states and (b) reflect the corresponding portion of thereceived light stream away from the sensing path when the mirror is in asecond of the two orientation states. Alternatively, the lightmodulating elements may be elements whose transmittances areindependently controllable (as in an LCD shutter array). Variousembodiments of the light modulating elements are contemplated.

The light sensing device may be configured to receive the modulatedlight stream and to capture (or generate) a sequence of samplesrepresenting intensity of the modulated light stream as a function oftime. These samples are the compressive imaging measurements. The actionof modulating the received light stream with the spatial patterns may becoordinated (synchronized) with the action of sampling the intensity ofthe modulated light stream. Alternatively, the two processes (modulationand sampling) may operate asynchronously.

In one embodiment of the system, calibration patterns are included inthe series of spatial patterns in addition to measurement patterns. Thecalibration patterns are designed so that the corresponding samples fromthe light sensing device represent samples of background light level.For example, the calibration patterns may be patterns that instruct allthe light modulating elements to take their maximum transfer state(e.g., instruct all mirrors to take the orientation state that reflectsonto the sensing path, or, instruct all LCD shutters to take theirmaximum transparency state.) A wide variety of possibilities areavailable for the structure of the calibration patterns.

The system may include a processing unit to separate the sequence ofsamples captured by the light sensing device into a first subsequence ofsamples that correspond to the measurement patterns and a secondsubsequence of samples that correspond to the calibration patterns. Thefirst subsequence of samples represent the compressive imagingmeasurements. The second subsequence of samples represents variation inthe background light level. The second subsequence may be used tocompensate the compressive imaging measurements (the first subsequenceof samples) for the variation in the background light level.

As described above, the light sensing device may be configured toreceive the modulated light stream and to capture a sequence of samplesrepresenting intensity of the modulated light stream. In the case wherethe light modulation unit is realized by the set of mirrors, themodulated light stream at any given time is made up of the lightportions that are reflected onto the sensing path by mirrors in thefirst orientation state. In this case, the samples captured by the lightsensing device represent a cumulative intensity of the light portionsreflected onto the sensing path as a function of time.

In one embodiment, the system may include the above-described set ofmirrors and light sensing device, and also include a second lightsensing device. The second light sensing device is configured to receivethe light portions that are reflected by mirrors in the secondorientation state and to generate a second sequence of samplesrepresenting a cumulative intensity of those light portions as afunction of time. At any given sampling time, each mirror may be thoughtof as being either in the first orientation state or the secondorientation state. Thus, sample sequences captured by the two lightsensing devices may be combined to infer the power (or intensity) of thelight stream incident upon the mirrors as a function of time. Thatcombined sequence represents variation in background light level(ambient illumination level). In this embodiment, the spatial patternssupplied to the mirrors need not include the calibration patterns of theembodiment described above.

In yet another embodiment, the system includes an optical subsystem(e.g., a beam splitter, a prism, etc.), a light modulation unit, a firstlight sensing device and a second light sensing device. The opticalsubsystem is configured to receive light from the environment and toseparate the received light into a first beam and a second beam.

The light modulation unit is configured to modulate the first beam witha series of spatial patterns to obtain a modulated beam. The first lightsensing device is configured to receive the modulated beam and togenerate a first sequence of samples representing intensity of themodulated beam as a function of time. The first sample sequencecorresponds to the compressive imaging measurements. The second lightsensing device is configured to generate a second sequence of samplesthat represents intensity of the second beam as a function of time. Thesecond sample sequence corresponds to variation in the background lightlevel.

In yet another embodiment, the background light level signal is derivedby low-pass filtering the compressive imaging measurements from thefirst light sensing device. Thus, the second light sensing device of thetwo-separate-beams system described immediately above may be omitted inthis embodiment.

In yet another embodiment, the compressive imaging measurements may behigh-pass filtered and then envelope detected to obtain an estimate ofthe background light level signal.

In yet another embodiment, the background light level signal is derivedby using a light power meter that has its own light input path, i.e.,separate from the input path that leads to the light modulation unit.

In yet another embodiment, the compressive imaging measurements may behigh-pass filtered in order to remove (or decrease or attenuate) theeffect of background light variation from the compressive imagingmeasurements.

Any two or more of the embodiments described herein may be used incombination. As one example, the above-described embodiment that usestwo light sensing devices to measure the light reflected in both mirrororientations may be combined with the low-pass filtering embodiment, inwhich case low-pass filtering could be performed on the measurementsfrom each light sensing device, thereby producing two background lightsignals. Those two signals may be combined (e.g., averages) to obtain arefined background light signal.

Various additional embodiments are described in U.S. ProvisionalApplication No. 61/372,826, which is hereby incorporated by reference inits entirety as though fully and completely set forth herein.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiments isconsidered in conjunction with the following drawings.

FIG. 1 illustrates a compressive sensing camera according to the priorart.

FIG. 2A illustrates one embodiment of a system 100 that is operable tocapture compressive imaging samples and also samples of background lightlevel. (LMU is an acronym for “light modulation unit”. MLS is an acronymfor “modulated light stream”. LSD is an acronym for “light sensingdevice”.)

FIG. 2B illustrates an embodiment of system 100 that includes aprocessing unit 150.

FIG. 2C illustrates an embodiment of system 100 that includes an opticalsubsystem 105 to focus received light L onto the light modulation unit110.

FIG. 2D illustrates an embodiment of system 100 that includes an opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 2E illustrates an embodiment where the optical subsystem 117 isrealized by a lens 117L.

FIG. 2F illustrates an embodiment of system 100 that includes a controlunit that is configured to supply a series of spatial patterns to thelight modulation unit 110.

FIG. 3A illustrates system 200, where the light modulation unit 110 isrealized by a plurality of mirrors (collectively referenced by label110M).

FIG. 3B shows an embodiment of system 200 that includes the processingunit 150.

FIG. 4 shows an embodiment of system 200 that includes the opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 5A shows an embodiment of system 200 where the optical subsystem117 is realized by the lens 117L.

FIG. 5B shows an embodiment of system 200 where the optical subsystem117 is realized by a mirror 117M and lens 117L in series.

FIG. 5C shows another embodiment of system 200 that includes a TIR prismpair 107.

FIG. 6A shows an embodiment of a system 600 that injects calibrationpatterns among the spatial patterns being fed to the light modulationunit 110 so that the samples captured by the light sensing device 130include calibration information in addition to compressive imagingmeasurements.

FIG. 6B shows an embodiment of system 600 that includes a control unit.The control unit supplies the spatial patterns to the light modulationunit 110.

FIG. 6C shows an embodiment of system 600 where the light modulationunit 110 is realized by the mirrors 110M.

FIG. 7A shows an embodiment of a method 700 for obtaining informationregarding background light variation by inserting calibration patternsamong the spatial patterns being fed to a light modulation unit.

FIG. 7B shows an embodiment of a method 750 for obtaining informationregarding background light variation by inserting calibration patternsamong the spatial patterns being fed to a set (e.g., array) ofcontrollable mirrors.

FIG. 8 shows an embodiment of a system 800 that measures the lightreflected at any given time from mirrors in the first orientation stateand the light reflected at any given time from mirrors in the secondorientation state. The two resulting signals may be combined to inferthe background light level as a function of time.

FIG. 9 illustrates an embodiment of system 800 that includes aprocessing unit and an optical subsystem 105 to focus incoming lightonto the mirrors 110M.

FIG. 10 illustrates an embodiment of system 800 that includes: anoptical subsystem 117 to direct or focus or concentrate light portionsreflected by the mirrors in the first orientation state onto a lightsensing surface of the light sensing device 130, and an opticalsubsystem 118 to direct or focus or concentrate light portions reflectedby the mirrors in the second orientation state onto a light sensingsurface of the light sensing device 135.

FIG. 11 illustrates one particular embodiment of optical subsystems 117and 118 using mirrors and lenses.

FIG. 12 illustrates an embodiment of method 1200 that determines orestimates background light level by combining measurements from twolight sensing devices, where a first of the two devices measures lightreflected from mirrors in a first orientation state and the second ofthe two light sensing devices measures light reflected from mirrors in asecond orientation state.

FIG. 13A illustrates an embodiment of system 1300 that separates theincoming light into a first stream and a second stream. The first streamproceeds to the light modulation unit 110. The intensity of the secondstream is measured and serves as an indicator of the background lightlevel.

FIG. 13B illustrates an embodiment of a system 1350 that includes anoptical subsystem 1360 that has the effect of separating the incidentlight into a first stream and a second stream, but the second stream isnot used.

FIG. 13C illustrates an embodiment of system 1350 where opticalsubsystem 1360 is realized by a TIR prism pair 107.

FIG. 14 illustrates an embodiment of system 1300 that includes thecontrol unit 120.

FIG. 15 illustrates an embodiment of system 1300 where the lightmodulation unit 110 is realized by the mirrors 110M.

FIG. 16A illustrates an embodiment of system 1300 that includes theprocessing unit 150.

FIG. 16B illustrates an embodiment of system 1300 that includes a beamsplitter that separates the incoming light into a transmitted beam and areflected beam.

FIG. 16C shows a special case of the system of FIG. 16B, where the lightmodulation unit 110 is realized by the mirrors 110M.

FIG. 16D shows an embodiment of system 1300 that includes a TIR prismpair 1310T that separates the incoming light into a transmitted beam anda reflected beam.

FIG. 16E shows an embodiment of system 1300 that includes an opticalsubsystem 117 to direct or focus or concentrate light onto the lightsensing device 130 and an optical subsystem 1315 to direct or focus orconcentrate light onto the light sensing device 1320.

FIG. 16F shows a special case of the system of FIG. 16E, where the lightmodulation unit 110 is realized by the mirrors 110M.

FIG. 17A illustrates an embodiment of a method 1700 that involvesseparating a received stream of light into a first stream and a secondstream. The first stream is used to obtain imaging measurements. Thesecond stream is measured in order to provide an indication of thebackground light level.

FIG. 17B illustrates an embodiment of a method 1750 that involvesseparating a received stream of light into a first stream and a secondstream. The first stream is used to obtain imaging measurements. Thesecond stream is measured in order to provide an indication of thebackground light level. This embodiment involves driving a plurality ofmirrors through a series of spatial patterns.

FIG. 18A illustrates an embodiment of system 1800 configured to derivean indication of background light level as a function of time byperforming low-pass filtering on the sequence of measurements capturedby the light sensing device 130.

FIG. 18B illustrates an embodiment of system 1800 that includes thecontrol unit 120.

FIG. 18C illustrates an embodiment of system 1800 that includes thecontrol unit 120 and the processing unit 150.

FIG. 19A illustrates an embodiment of a method 1900 that obtains anindication of background light level as a function of time by performinglow-pass filtering on a sequence of compressive imaging measurements.

FIG. 19B illustrates an embodiment of a method 1950 that obtains anindication of background light level as a function of time by performinglow-pass filtering on a sequence of compressive imaging measurements.This embodiment involves driving a plurality of mirrors through a seriesof spatial patterns.

FIG. 19C illustrates an embodiment of a method 1972 that obtains anindication of background light level as a function of time by performinghigh-pass filtering followed by envelope detection on a compressiveimaging signal.

FIG. 19D illustrates an embodiment of a method 1980 that obtains anindication of background light level as a function of time by performinghigh-pass filtering followed by envelope detection on a sequence ofcompressive imaging measurements.

FIG. 19E illustrates an embodiment of a system 1988 that extractsinformation and removes (or decreases or attenuates) the effect ofbackground light variation from a compressive imaging signal.

FIG. 19F illustrates an embodiment of a method 1990 for extractinginformation and removing (or decreasing or attenuating) the effect ofbackground light variation from a compressive imaging signal.

FIG. 20A illustrates an embodiment of a system 2000 that includes twoseparate input paths for light. The system captures compressive imagingmeasurements from light entering through the first path, and capturesmeasurements of power of the light entering through the second path. Thelatter measurements indicate background light level as function of time.

FIG. 20B illustrates an embodiment of system 2000 that includes thecontrol unit 120.

FIG. 20C illustrates an embodiment of system 2000 that includes theprocessing unit 150.

FIG. 21 illustrates an embodiment of system 2000, where the lightmodulation unit 110 is realized by the mirrors 110M.

FIG. 22 illustrates an embodiment of system 2000 that includes twoseparately-housed units, one of the units including the light modulationunit 110 and the light sensing device 130, and the other unit includingthe light power meter 2010.

FIG. 23A illustrates an embodiment of a method 2300 for compensating asequence of compressive imaging measurements based on power measurementsupplied by a light power meter.

FIG. 23B illustrates an embodiment of a method 2350 for compensating asequence of compressive imaging measurements based on power measurementssupplied by a light power meter. This embodiment involves driving aplurality of mirrors through a series of spatial patterns.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following patent applications provide teachings regardingcompressive sensing and compressive imaging:

-   U.S. Provisional Application No. 60/673,364 entitled “Method and    Apparatus for Optical Image Compression,” filed on Apr. 21, 2005;-   U.S. Provisional Application No. 60/679,237 entitled “Method and    Apparatus for Reconstructing Data from Multiple Sources,” filed on    May 10, 2005;-   U.S. Provisional Application No. 60/729,983 entitled “Random Filters    for Compressive Sampling and Reconstruction,” filed on Oct. 25,    2005;-   U.S. Provisional Application No. 60/732,374 entitled “Method and    Apparatus for Compressive Sensing for Analog-to-Information    Conversion,” filed on Nov. 1, 2005;-   U.S. Provisional Application No. 60/735,616 entitled “Method and    Apparatus for Distributed Compressed Sensing,” filed on Nov. 10,    2005;-   U.S. Provisional Application No. 60/759,394 entitled “Sudocodes:    Efficient Compressive Sampling Algorithms for Sparse Signals,” filed    on Jan. 16, 2006;-   U.S. patent application Ser. No. 11/379,688 entitled “Method and    Apparatus for Compressive Imaging Device”, filed on Apr. 21, 2006;    and-   U.S. patent application Ser. No. 12/791,171 entitled “Method and    Apparatus for Compressive Imaging Device”, filed on Jun. 1, 2010.

Terminology

A memory medium is a non-transitory medium configured for the storageand retrieval of information. Examples of memory media include: variouskinds of semiconductor-based memory such as RAM and ROM; various kindsof magnetic media such as magnetic disk, tape, strip and film; variouskinds of optical media such as CD-ROM and DVD-ROM; various media basedon the storage of electrical charge and/or any of a wide variety ofother physical quantities; media fabricated using various lithographictechniques; etc. The term “memory medium” includes within its scope ofmeaning the possibility that a given memory medium might be a union oftwo or more memory media that reside at different locations, e.g., ondifferent chips in a system or on different computers in a network. Amemory medium is typically computer-readable, e.g., is capable of beingread by a computer.

A computer-readable memory medium may be configured so that it storesprogram instructions and/or data, where the program instructions, ifexecuted by a computer system, cause the computer system to perform amethod, e.g., any of a method embodiments described herein, or, anycombination of the method embodiments described herein, or, any subsetof any of the method embodiments described herein, or, any combinationof such subsets.

A computer system is any device (or combination of devices) having atleast one processor that is configured to execute program instructionsstored on a memory medium. Examples of computer systems include personalcomputers (PCs), workstations, laptop computers, tablet computers,mainframe computers, server computers, client computers, network orInternet appliances, hand-held devices, mobile devices, personal digitalassistants (PDAs), computer-based television systems, grid computingsystems, wearable computers, computers implanted in living organisms,computers embedded in head-mounted displays, computers embedded insensors forming a distributed network, etc.

A programmable hardware element (PHE) is a hardware device that includesmultiple programmable function blocks connected via a system ofprogrammable interconnects. Examples of PHEs include FPGAs (FieldProgrammable Gate Arrays), PLDs (Programmable Logic Devices), FPOAs(Field Programmable Object Arrays), and CPLDs (Complex PLDs). Theprogrammable function blocks may range from fine grained (combinatoriallogic or look up tables) to coarse grained (arithmetic logic units orprocessor cores).

As used herein, the term “light” is meant to encompass within its scopeof meaning any electromagnetic radiation whose spectrum lies within thewavelength range [λ_(L), λ_(U)], where the wavelength range includes thevisible spectrum, the ultra-violet (UV) spectrum, infrared (IR) spectrumand the terahertz (THz) spectrum. Thus, for example, visible radiation,or UV radiation, or IR radiation, or THz radiation, or any combinationthereof is “light” as used herein.

In some embodiments, a computer system may be configured to include aprocessor (or a set of processors) and a memory medium, where the memorymedium stores program instructions, where the processor is configured toread and execute the program instructions stored in the memory medium,where the program instructions are executable by the processor toimplement a method, e.g., any of the various method embodimentsdescribed herein, or, any combination of the method embodimentsdescribed herein, or, any subset of any of the method embodimentsdescribed herein, or, any combination of such subsets.

System 100 for Operating on Light

A system 100 for operating on light may be configured as shown in FIG.2A.

The system 100 may include a light modulation unit 110 and a lightsensing device 130. In some embodiments, the system 100 may also includea mechanism 140 for measuring background light level. The mechanism 140is represented abstractly because it may take different forms in thedifferent ones of the embodiments described below.

The light modulation unit 110 is configured to modulate a receivedstream of light L with a series of spatial patterns in order to producea modulated light stream (MLS). The spatial patterns of the series maybe applied sequentially to the light stream so that successive timeslices of the light stream are modulated, respectively, with successiveones of the spatial patterns. (The action of sequentially modulating thelight stream L with the spatial patterns imposes the structure of timeslices on the light stream.) The light modulation unit 110 includes aplurality of light modulating elements configured to modulatecorresponding portions of the light stream. Each of the spatial patternsspecifies an amount (or extent or value) of modulation for each of thelight modulating elements. Mathematically, one might think of the lightmodulation unit's action of applying a given spatial pattern asperforming an element-wise multiplication of a light field vector(x_(ij)) representing a time slice of the light stream L by a vector ofscalar modulation values (m_(ij)) to obtain a time slice of themodulated light stream: (m_(ij))*(x_(ij))=(m_(ij)*x_(ij)). The vector(m_(ij)) is specified by the spatial pattern. Each light modulatingelement effectively scales (multiplies) the intensity of itscorresponding stream portion by the corresponding scalar factor.

The light modulation unit 110 may be realized in various ways. In someembodiments, the LMU 110 may be realized by a plurality of mirrors(e.g., micromirrors) whose orientations are independently controllable.In another set of embodiments, the LMU 110 may be realized by an arrayof elements whose transmittances are independently controllable, e.g.,as with an array of LCD shutters. An electrical control signal suppliedto each element controls the extent to which light is able to transmitthrough the element. In yet another set of embodiments, the LMU 110 maybe realized by an array of independently-controllable mechanicalshutters (e.g., micromechanical shutters) that cover an array ofapertures, with the shutters opening and closing in response toelectrical control signals, thereby controlling the flow of lightthrough the corresponding apertures. In yet another set of embodiments,the LMU 110 may be realized by a perforated mechanical plate, with theentire plate moving in response to electrical control signals, therebycontrolling the flow of light through the corresponding perforations. Inyet another set of embodiments, the LMU 110 may be realized by an arrayof transceiver elements, where each element receives and thenretransmits light in a controllable fashion. In yet another set ofembodiments, the LMU 110 may be realized by a grating light valve (GLV)device. In yet another embodiment, the LMU 110 may be realized by aliquid-crystal-on-silicon (LCOS) device.

In some embodiments, the light modulating elements are arranged in anarray, e.g., a two-dimensional array or a one-dimensional array. Any ofvarious array geometries are contemplated. For example, in someembodiments, the array is a square array or rectangular array. Inanother embodiment, the array is hexagonal. In some embodiments, thelight modulating elements are arranged in a spatially random fashion.

The light sensing device 130 is configured to receive the modulatedlight stream MLS and to generate a sequence of samples {I_(MLS)(k)}representing intensity of the modulated light stream as a function oftime. Each of the samples may be interpreted as an inner product betweena corresponding time slice of the light stream L and a corresponding oneof the spatial patterns. The set of samples {I_(MLS)(k)} comprises anencoded representation, e.g., a compressed representation, of an image(or a video sequence) and may be used to construct the image (or videosequence). (For video sequence construction, the samples may bepartitioned into contiguous subsets, and then the subsets may beprocessed to construct corresponding images.)

In some embodiments, the sample set {I_(MLS)(k)} may be used for somepurpose other than or in addition to image (or video) construction. Forexample, system 100 (or some other system) may operate on the sample setor a plurality of such sets to perform an inference task, such asdetecting the presence of a signal or object, classifying a signal or anobject, estimating one or more parameters relating to a signal or anobject, tracking a signal or an object, etc.

The light sensing device 130 may include one or more light sensingelements and one or more A/D converters. The term “light sensingelement” may be interpreted as meaning “a transducer between a lightsignal and an electrical signal”. For example, a photodiode may beinterpreted as a light sensing element. In various other embodiments,light sensing elements might include devices such asmetal-semiconductor-metal (MSM) photodetectors, phototransistors,phototubes and photomultiplier tubes.

In some embodiments, the light sensing device 130 includes exactly onelight sensing element. For example, the single light sensing element maybe a photodiode. The light sensing element may convert light impingingon its light sensing surface into an analog electrical signalrepresenting intensity (or power) of the impinging light as a functionof time. The analog electrical signal may be digitized by an A/Dconverter to generate the sequence of samples {I_(MLS)(k)}. In someembodiments, the light sensing device 130 may also include an amplifier,e.g., a transimpedance amplifier (TIA), to amplify the analog electricalsignal prior to A/D conversion.

In some embodiments, the light sensing device 130 may include aplurality of light sensing elements (e.g., photodiodes). Each lightsensing element may convert light impinging on its light sensing surfaceinto a corresponding analog electrical signal representing intensity ofthe impinging light as a function of time. In some embodiments, eachlight sensing element may couple to a corresponding amplifier so thatthe analog electrical signal produced by the light sensing element canbe amplified prior to digitization. System 100 may be configured so thateach light sensing element receives, e.g., a corresponding spatialportion (or spectral portion) of the modulated light stream.

In one embodiment, the analog electrical signals produced, respectively,by the light sensing elements may be summed to obtain a sum signal. Thesum signal may then be digitized by an A/D converter to obtain thesequence of samples {I_(MLS)(k)}. In another embodiment, the analogelectrical signals may be individually digitized, each with its own A/Dconverter, to obtain corresponding sample sequences. The samplesequences may then be added to obtain the sequence {I_(MLS)(k)}. Inanother embodiment, the analog electrical signals produced by the lightsensing elements may be sampled by a smaller number of A/D convertersthan light sensing elements through the use of time multiplexing. Forexample, in one embodiment, light sensing device 130 may be configuredto sample two or more of the analog electrical signals by switching theinput of an A/D converter among the outputs of the two or morecorresponding light sensing elements at a sufficiently high rate.

In some embodiments, the light sensing device 130 may include an arrayof light sensing elements. Arrays of any of a wide variety of sizes andconfigurations are contemplated. In one embodiment, the light sensingdevice 130 includes a focal plane array coupled to a readout integratedcircuit. In one embodiment, the light sensing device 130 may include anarray of cells, where each cell is configured to integrate and holdphoto-induced charge created by a corresponding one of the light sensingelements, and to convert the integrated charge into a corresponding cellvoltage. The light sensing device may also include circuitry configuredto sample the cell voltages using one or more A/D converters.

In some embodiments, the light sensing device 130 may include aplurality (or array) of light sensing elements, where each light sensingelement is configured to receive a corresponding spatial portion of themodulated light stream, and each spatial portion of the modulated lightstream comes from a corresponding sub-region of the array of lightmodulating elements. (For example, the light sensing device 130 mayinclude a quadrant photodiode, where each quadrant of the photodiode isconfigured to receive modulated light from a corresponding quadrant ofthe array of light modulating elements. As another example, the lightsensing element 130 may include a bi-cell photodiode.) Each lightsensing element generates a corresponding signal representing intensityof the corresponding spatial portion as a function of time. Each signalmay be digitized (e.g., by a corresponding A/D converter) to obtain acorresponding sequence of samples. Each sequence of samples may beprocessed to recover a corresponding sub-image. The sub-images may bejoined together to form a whole image. Any of the various embodimentsdescribed herein for estimating or determining background light level asa function of time may be implemented for each of the sample sequences.Furthermore, any of the various embodiments described herein forcompensating for variation in background light level may be implementedfor each of the sample sequences.

In some embodiments, the light sensing device 130 includes a smallnumber of light sensing elements (e.g., in respective embodiments, one,two, less than 8, less than 16, less the 32, less than 64, less than128, less than 256). Because the light sensing device of theseembodiments includes a small number of light sensing elements (e.g., farless than the typical modern CCD-based or CMOS-based camera), an entityconfiguring any of these embodiments may afford to spend more per lightsensing element to realize features that are beyond the capabilities ofmodern array-based image sensors of large pixel count, e.g., featuressuch as higher sensitivity, extended range of sensitivity, new range(s)of sensitivity, extended dynamic range, higher bandwidth/lower responsetime. Furthermore, because the light sensing device includes a smallnumber of light sensing elements, an entity configuring any of theseembodiments may use newer light sensing technologies (e.g., based on newmaterials or combinations of materials) that are not yet mature enoughto be manufactured into focal plane arrays (FPA) with large pixel count.For example, new detector materials such as super-lattices, quantumdots, carbon nanotubes and graphene can significantly enhance theperformance of IR detectors by reducing detector noise, increasingsensitivity, and/or decreasing detector cooling requirements.

In one embodiment, the light sensing device 130 is a thermo-electricallycooled InGaAs detector. (InGaAs stands for “Indium Gallium Arsenide”.)In other embodiments, the InGaAs detector may be cooled by othermechanisms (e.g., liquid nitrogen or a Sterling engine). In yet otherembodiments, the InGaAs detector may operate without cooling. In yetother embodiments, different detector materials may be used, e.g.,materials such as MCT (mercury-cadmium-telluride), InSb (IndiumAntimonide) and VOx (Vanadium Oxide).

In different embodiments, the light sensing device 130 may be sensitiveto light at different wavelengths or wavelength ranges. In someembodiments, the light sensing device 130 may be sensitive to light overa broad range of wavelengths, e.g., over the entire visible spectrum orover the entire range [λ_(L),λ_(U)] as defined above.

In some embodiments, the light sensing device 130 may include one ormore dual-sandwich photodetectors. A dual sandwich photodetectorincludes two photodiodes stacked (or layered) one on top of the other.

In one embodiment, the light sensing device 130 may include one or moreavalanche photodiodes.

In some embodiments, a filter may be placed in front of the lightsensing device 130 to restrict the modulated light stream to a specificrange of wavelengths or polarization. Thus, samples captured by thelight sensing device 130 may be representative of the intensity of therestricted light stream. For example, by using a filter that passes onlyIR light, the light sensing device may be effectively converted into anIR detector. The sample principle may be applied to effectively convertthe light sensing device into a detector for red or blue or green or UVlight or IR light, or, a detector for light of a certain polarization.

In some embodiments, system 100 includes a color wheel whose rotation issynchronized with the application of the spatial patterns to the lightmodulation unit. As it rotates, the color wheel cyclically applies anumber of optical bandpass filters to the modulated light stream MLS.Each bandpass filter restricts the modulated light stream to acorresponding sub-band of wavelengths. Thus, the samples captured by thelight sensing device 130 will include samples of intensity in each ofthe sub-bands. The samples may be de-multiplexed to form separatesub-band sequences. Each sub-band sequence may be processed to generatea corresponding sub-band image. (As an example, the color wheel mayinclude a red-pass filter, a green-pass filter and a blue-pass filter tosupport color imaging.)

The mechanism 140 is responsible for generating a background lightsignal, e.g., a sequence of samples that represents variation inbackground light level (i.e., ambient illumination level). As notedabove, the various ways the mechanism may be achieved will be discussedin the various embodiments below.

In some embodiments, the system 100 may include a memory (or a set ofmemories of one or more kinds).

In some embodiments, system 100 may include a processing unit 150, e.g.,as shown in FIG. 2B. The processing unit 150 may be a digital circuit ora combination of digital circuits. For example, the processing unit maybe a microprocessor (or system of interconnected of microprocessors), aprogrammable hardware element such as a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), or anycombination such elements. The processing unit 150 may be configured toperform one or more functions such as compensation of the capturedsamples {I_(MLS)(k)}, image/video construction, system control, userinterface, statistical analysis, one or more inferences tasks. Theprocessing unit is not necessarily distinct from the mechanism 140 formeasuring background light level. In some embodiments, they may beidentical or overlap to some extent.

The system 100 (e.g., the processing unit 150) may store the samples{I_(MLS)(k)} and the samples of background light level in a memory,e.g., a memory resident in the system 100 or in some other system. Thesamples {I_(MLS)(k)} and the samples of background light level areusable to construct an image (or video sequence).

The samples of the background light level may be used to compensate thesamples {I_(MLS)(k)} for the variation in the background light level,e.g., in order to approximate the hypothetical samples that would havebeen captured at the light sensing device if the background light levelhad remained constant, e.g., at some average level. A wide variety ofcomputational algorithms or formulas for performing the compensation arecontemplated.

In some embodiments, compensation implies increasing the value of asample I_(MLS)(k) when the corresponding sample of background lightlevel is small, and decreasing the value of the sample I_(MLS)(k) whenthe corresponding sample of the background light level is large.

In some embodiments, the compensation may be performed in accord withthe following expression:I _(CMPN)(k)=I _(MLS)(k)*A/I _(BGL)(k),where {I_(BGL)(k)} denotes the samples of the background light level,where A is a constant, where {I_(CMPN)(k)} denotes the compensatedsamples. In one embodiment, the constant A may be the average of{I_(BGL)(k)} over the image/video capture interval or some subintervalthereof.

In other embodiments, the compensation may be performed based on theexpressionI _(CMPN)(k)=I _(MLS)(k)−I _(BGL)(k),or more generally, on the expressionI _(CMPN)(k)=α*I _(MLS)(k)−β*I _(BGL)(k),where α and β are scaling factors. These expressions may be useful insituations where the background light can be modeled as an additivedisturbance to an information signal.

The action of compensating the samples {I_(MLS)(k)} for the variation inbackground light level may be performed by system 100 (e.g., byprocessing unit 150). The compensated samples may be stored in a memory,e.g., a memory resident in system 100 or some other memory. Thecompensated samples are useable to construct an image or video sequence,e.g., based on any construction algorithm known in the field ofcompressive sensing. In some embodiments, the compensated samples may beused for some purpose other than or in addition to image (or video)construction. For example, system 100 (or some other system) may operateon the compensated samples to perform an inference task, such asdetecting the presence of a signal or object, identifying a signal or anobject, classifying a signal or an object, estimating one or moreparameters relating to a signal or an object, tracking a signal or anobject, etc. For example, an object under observation by system 100 maybe identified or classified by virtue of its sample set {I_(MLS)(k)} orparameters derived from that sample set being similar to one of acollection of stored sample sets (or parameter sets).

In one embodiment, processing unit 150 is configured to operate on thecompensated samples to generate the image or video sequence. In thisembodiment, the processing unit 150 may include a microprocessorconfigured to execute software (i.e., program instructions), especiallysoftware for performing an image/video construction algorithm. In oneembodiment, system 100 is configured to transmit the compensated samples(and perhaps also the random seed(s) used to generate the spatialpatterns) to some other system through a communication channel. Thatother system may operate on the compensated samples to construct theimage/video. System 100 may have one or more interfaces configured forsending (and perhaps also receiving) data through one or morecommunication channels, e.g., channels such as wireless channels, wiredchannels, fiber optic channels, acoustic channels, laser-based channels,etc.

In one embodiment, system 100 may include a light transmitter configuredto generate a light beam (e.g., a laser beam), to modulate the lightbeam with a data signal and to transmit the modulated light beam intospace or onto an optical fiber. System 100 may also include a lightreceiver configured to receive a modulated light beam from space or froman optical fiber, and to recover a data stream from the modulated lightbeam.

In some embodiments, the samples {I_(BGL)(k)} of background light levelmay be examined to determine any samples that depart from the mean valueof background light level (over the measurement period) by more than athreshold amount, and then discard the corresponding samples from thesequence {I_(MLS)(k)}. Thus, the remaining samples of the sequence{I_(MLS)(k)} can be counted upon to conform to a sufficiently smallvariation of background light level around the mean. These remainingsamples may be used to construct the image or video sequence. Theconstruction algorithm may be provided with a list of the time positionsof the samples that were discarded.

In one embodiment, system 100 may be configured as a low-cost sensorsystem having minimal processing resources, e.g., processing resourcessufficient to perform the background light level compensation but notenough to perform image (or video) construction in user-acceptable time.In this embodiment, the system 100 may store and/or transmit thecompensated samples {I_(CMPN)(k)} so that another agent, moreplentifully endowed with processing resources, may perform theimage/video construction based on the compensated samples.

In one embodiment, the action of compensating the samples {I_(MLS)(k)}for the variation in background light level may be performed by somesystem other than system 100. In this case, system 100 may transmit thesamples {I_(MLS)(k)} and the samples of background light level to theother system through a communication channel. The other system may usethe compensated samples to construct the image/video.

In some embodiments, system 100 may include an optical subsystem 105that is configured to modify or condition the light stream L before itarrives at the light modulation unit 110, e.g., as shown in FIG. 2C. Forexample, the optical subsystem 105 may be configured to receive thelight stream L from the environment and to focus the light stream onto amodulating plane of the light modulation unit 110. The optical subsystem105 may include a camera lens (or set of lenses). The lens (or set oflenses) may be adjustable to accommodate a range of distances toexternal objects being imaged/sensed/captured.

In some embodiments, system 100 may include an optical subsystem 117 todirect the modulated light stream MLS onto a light sensing surface (orsurfaces) of the light sensing device 130.

In some embodiments, the optical subsystem 117 is configured to focusthe modulated light stream onto the light sensing surface (or surfaces).The term “focus” implies an attempt to achieve the condition that rays(photons) diverging from a point on an object plane converge to a point(or an acceptably small spot) on an image plane. The term “focus” alsotypically implies continuity between the object plane point and theimage plane point (or image plane spot)—points close together on theobject plane map respectively to points (or spots) close together on theimage plane. In some embodiments of system 100 that include an array oflight sensing elements, it may be desirable for the modulated lightstream MLS to be focused onto the light sensing array so that there iscontinuity between points on the light modulation unit LMU and points(or spots) on the light sensing array.

In some embodiments, the optical subsystem 117 may be configured todirect the modulated light stream MLS onto the light sensing surface (orsurfaces) of the light sensing device 130 in a non-focusing fashion. Forexample, in some of the embodiments where the light sensing device 130includes only one photodiode, it may not be so important to achieve the“in focus” condition at the light sensing surface of the photodiodesince positional information of photons arriving at that light sensingsurface will be immediately lost.

In one embodiment, the optical subsystem 117 may be configured toreceive the modulated light stream and to concentrate the modulatedlight stream into an area (e.g., a small area) on a light sensingsurface of the light sensing device 130. Thus, the diameter of themodulated light stream may be reduced (possibly, radically reduced) inits transit from the optical subsystem 117 to the light sensing surface(or surfaces) of the light sensing device 130. For example, in someembodiments, the diameter may be reduced by a factor of more than 1.5to 1. In other embodiments, the diameter may be reduced by a factor ofmore than 2 to 1. In yet other embodiments, the diameter may be reducedby a factor of more than 10 to 1. In yet other embodiments, the diametermay be reduced by factor of more than 100 to 1. In yet otherembodiments, the diameter may be reduced by factor of more than 400to 1. In one embodiment, the diameter is reduced so that the modulatedlight stream is concentrated onto the light sensing surface of a singlelight sensing element (e.g., a single photodiode).

In some embodiments, this feature of concentrating the modulated lightstream onto the light sensing surface (or surfaces) of the light sensingdevice allows the light sensing device to sense at any given time thesum (or surface integral) of the intensities of the modulated lightportions within the modulated light stream. (Each time slice of themodulated light stream comprises a spatial ensemble of modulated lightportions due to the modulation unit's action of applying thecorresponding spatial pattern to the light stream.)

In some embodiments, the modulated light stream MLS may be directed ontothe light sensing surface of the light sensing device 130 withoutconcentration, i.e., without decrease in diameter of the modulated lightstream, e.g., by use of photodiode having a large light sensing surface,large enough to contain the cross section of the modulated light streamwithout the modulated light stream being concentrated.

In some embodiments, the optical subsystem 117 may include a lens 117L,e.g., as shown in FIG. 2E. In one embodiment, the lens 117L may be abiconvex lens or a condenser lens.

In some embodiments, the optical subsystem 117 may include one or moremirrors. In one embodiment, the optical subsystem 117 includes aparabolic mirror (or spherical mirror) to concentrate the modulatedlight stream onto a neighborhood (e.g., a small neighborhood) of theparabolic focal point. In this embodiment, the light sensing surface ofthe light sensing device may be positioned at the focal point.

In some embodiments, system 100 may include an optical mechanism (e.g.,an optical mechanism including one or more prisms and/or one or morediffraction gratings) for splitting or separating the modulated lightstream MLS into two or more separate streams (perhaps numerous streams),where each of the streams is confined to a different wavelength range.The separate streams may each be sensed by a separate light sensingdevice. (In some embodiments, the number of wavelength ranges may be,e.g., greater than 8, or greater than 16, or greater than 64, or greaterthan 256, or greater than 1024.) Furthermore, each separate stream maybe directed (e.g., focused or concentrated) onto the corresponding lightsensing device as described above in connection with optical subsystem117. The samples captured by each light sensing device may be used toconstruct a corresponding image for the corresponding wavelength range.In one embodiment, the modulated light stream is separated into red,green and blue streams to support color (R,G,B) measurements. In anotherembodiment, the modulated light stream may be separated into IR, red,green, blue and UV streams to support five-channel multi-spectralimaging: (IR, R, G, B, UV). In some embodiments, the modulated lightstream may be separated into a number of sub-bands (e.g., adjacentsub-bands) within the IR band to support multi-spectral orhyper-spectral IR imaging. In some embodiments, the number of IRsub-bands may be, e.g., greater than 8, or greater than 16, or greaterthan 64, or greater than 256, or greater than 1024. In some embodiments,the optical mechanism may include two or more stages of spectralseparation. For example, in a first stage the modulated light stream maybe separated into an IR stream confined to the IR band and one or moreadditional streams confined to other bands. In a second stage, the IRstream may be separated into a number of sub-bands (e.g., numeroussub-bands) (e.g., adjacent sub-bands) within the IR band to supportmultispectral or hyper-spectral IR imaging.

In some embodiments, system 100 may include an optical mechanism (e.g.,a beam splitter) for splitting or separating the modulated light streamMLS into two or more separate streams, e.g., where each of the streamshave the same (or approximately the same) spectral characteristics orwavelength range. The separate streams may then pass through respectivebandpass filters to obtain corresponding modified streams, wherein eachmodified stream is restricted to a corresponding band of wavelengths.Each of the modified streams may be sensed by a separate light sensingdevice. (In some embodiments, the number of wavelength bands may be,e.g., greater than 8, or greater than 16, or greater than 64, or greaterthan 256, or greater than 1024.) Furthermore, each of the modifiedstreams may be directed (e.g., focused or concentrated) onto thecorresponding light sensing device as described above in connection withoptical subsystem 117. The samples captured by each light sensing devicemay be used to construct a corresponding image for the correspondingwavelength band. In one embodiment, the modulated light stream isseparated into three streams which are then filtered, respectively, witha red-pass filter, a green-pass filter and a blue-pass filter. Theresulting red, green and blue streams are then respectively detected bythree light sensing devices to support color (R,G,B) acquisition. Inanother similar embodiment, five streams are generated, filtered withfive respective filters, and then measured to support (IR, R, G, B, UV)multi-spectral acquisition. In yet another embodiment, the modulatedlight stream of a given band may be separated into a number of (e.g.,numerous) sub-bands to support multi-spectral or hyper-spectral imaging.

In some embodiments, system 100 may include an optical mechanism forsplitting or separating the modulated light stream MLS into two or moreseparate streams. The separate streams may be directed to (e.g.,concentrated onto) respective light sensing devices. The light sensingdevices may be configured to be sensitive in different wavelengthranges, e.g., by virtue of their different material properties. Thesamples captured by each light sensing device may be used to construct acorresponding image for the corresponding wavelength range.

In some embodiments, system 100 may include a control unit 120configured to supply the spatial patterns to the light modulation unit110, as shown in FIG. 2F. The control unit may itself generate thepatterns or may receive the patterns from some other agent. The controlunit 120 and the light sensing device 130 may be controlled by a commonclock signal so that the light sensing device 130 can coordinate(synchronize) its action of capturing the samples {I_(MLS)(k)} with thecontrol unit's action of supplying spatial patterns to the lightmodulation unit 110. (System 100 may include clock generationcircuitry.)

In some embodiment, the control unit 120 may supply the spatial patternsto the light modulation unit in a periodic fashion.

The control unit 120 may be a digital circuit or a combination ofdigital circuits. For example, the control unit may include amicroprocessor (or system of interconnected of microprocessors), aprogrammable hardware element such as a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), or anycombination such elements.

In some embodiments, the control unit 120 may include a random numbergenerator (RNG) or a set of random number generators to generate thespatial patterns or some subset of the spatial patterns.

In some embodiments, system 100 is battery powered. In some embodiments,the system 100 includes a set of one or more solar cells and associatedcircuitry to derive power from sunlight.

In some embodiments, system 100 includes its own light source forilluminating the environment or a target portion of the environment.

In some embodiments, system 100 may include a display (or an interfaceconfigured for coupling to a display) for displaying constructedimages/videos.

Regarding the spatial patterns that are used to modulate the lightstream L, it should be understood that there are a wide variety ofpossibilities. In some embodiments, the control unit 120 may beprogrammable so that any desired set of spatial patterns may be used.

In some embodiments, the spatial patterns are binary valued. Such anembodiment may be used, e.g., when the light modulating elements aretwo-state devices. In some embodiments, the spatial patterns are n-statevalued, where each element of each pattern takes one of n states, wheren is an integer greater than two. (Such an embodiment may be used, e.g.,when the light modulating elements are each able to achieve n or moremodulation states). In some embodiments, the spatial patterns are realvalued, e.g., when each of the light modulating elements admits acontinuous range of modulation. (It is noted that even a two-statemodulating element may be made to effectively apply a continuous rangeof modulation by duty cycling the two states during modulationintervals.)

The spatial patterns may belong to a set of measurement vectors that isincoherent with a set of vectors in which the image/video isapproximately sparse (“the sparsity vector set”). (See “Sparse SignalDetection from Incoherent Projections”, Proc. Int. Conf. Acoustics,Speech Signal Processing—ICASSP, May 2006, Duarte et al.) Given two setsof vectors A={a_(i)} and B={b_(i)} in the same N-dimensional space, Aand B are said to be incoherent if their coherence measure μ(A,B) issufficiently small. The coherence measure is defined as:

${\mu\left( {A,B} \right)} = {\max\limits_{i,j}{{\left\langle {a_{i},b_{j}} \right\rangle }.}}$The number of compressive sensing measurements (i.e., samples of thesequence {I_(MLS)(k)} needed to construct an N-pixel image (or N-voxelvideo sequence) that accurately represents the scene being captured is astrictly increasing function of the coherence between the measurementvector set and the sparsity vector set. Thus, better compression can beachieved with smaller values of the coherence.

In some embodiments, the measurement vector set may be based on a code.Any of various codes from information theory may be used, e.g., codessuch as exponentiated Kerdock codes, exponentiated Delsarte-Goethalscodes, run-length limited codes, LDPC codes, Reed Solomon codes and ReedMuller codes.

In some embodiments, the measurement vector set corresponds to apermuted basis such as a permuted DCT basis or a permuted Walsh-Hadamardbasis, etc.

In some embodiments, the spatial patterns may be random or pseudo-randompatterns, e.g., generated according to a random number generation (RNG)algorithm using one or more seeds. In some embodiments, the elements ofeach pattern are generated by a series of Bernoulli trials, where eachtrial has a probability p of giving the value one and probability 1−p ofgiving the value zero. (For example, in one embodiment p=1/2.) In someembodiments, the elements of each pattern are generated by a series ofdraws from a Gaussian random variable.)

The system 100 may be configured to operate in a compressive fashion,where the number of the samples {I_(MLS)(k)} captured by the system 100is less than (e.g., much less than) the number of pixels in the image(or video) to be constructed from the samples. In many applications,this compressive realization is very desirable because it saves on powerconsumption, memory utilization and transmission bandwidth consumption.However, non-compressive realizations are contemplated as well.

In some embodiments, the system 100 is configured as a camera thatcaptures information representing an image (or a series of images) ofsome external object or scene. The camera system may take differentforms in different applications domains, e.g., domains such as visiblelight photography, infrared photography, ultraviolet photography,high-speed photography, low-light photography, underwater photography,multi-spectral imaging, hyper-spectral imaging, etc. In someembodiments, system 100 is configured to operate in conjunction with (oras part of) another system, e.g., in conjunction with a microscope, atelescope, a robot, a security system, a surveillance system, a node ina distributed sensor network, etc.

In some embodiments, system 100 may also be configured to operate as aprojector. Thus, system 100 may include a light source, e.g., a lightsource located at or near a focal point of optical subsystem 117. Thelight modulation unit 110 may be supplied with an image (or a videosequence) so that the image (or video sequence) can be displayed on adisplay surface (e.g., screen).

In some embodiments, the background light signal generated by mechanism140 may be used to control the size (e.g., diameter) of a physicalaperture that is applied to the received light stream L, to control theamount of light received by system 100. Alternatively, the backgroundlight signal may be used to control the size (e.g., diameter) of anaperture implemented by the light modulation unit, by appropriatecontrol of the spatial patterns. Prior to being applied to the lightmodulation unit, each of the spatial patterns may be masked with anaperture pattern that nullifies the values of the spatial patternoutside a central region (e.g., central disk).

In one realization 200 of system 100, the light modulation unit 110 maybe realized by a plurality of mirrors, e.g., as shown in FIG. 3A. (Themirrors are collectively indicated by the label 110M.) The mirrors 110Mare configured to receive corresponding portions of the light L receivedfrom the environment, albeit not necessarily directly from theenvironment. (There may be one or more optical elements, e.g., one ormore lenses along the input path to the mirrors 110M.) Each of themirrors is configured to controllably switch between two orientationstates. In addition, each of the mirrors is configured to (a) reflectthe corresponding portion of the light onto a sensing path 115 when themirror is in a first of the two orientation states and (b) reflect thecorresponding portion of the light away from the sensing path when themirror is in a second of the two orientation states.

In some embodiments, the mirrors 110M are arranged in an array, e.g., atwo-dimensional array or a one-dimensional array. Any of various arraygeometries are contemplated. For example, in one embodiment, the arrayis a square array. In another embodiment, the array is hexagonal. Insome embodiments, the mirrors are arranged in spatially random fashion.

The mirrors 110M may be part of a digital micromirror device (DMD). Forexample, in some embodiments, one of the DMDs manufactured by TexasInstruments may be used.

The control unit 120 may be configured to drive the orientation statesof the mirrors through the series of spatial patterns, where each of thepatterns of the series specifies an orientation state for each of themirrors.

The light sensing device 130 may be configured to receive the lightportions reflected at any given time onto the sensing path 115 by thesubset of mirrors in the first orientation state and to generate asequence of samples representing a cumulative intensity of the receivedlight portions as a function of time. As the mirrors are driven throughthe series of spatial patterns, the subset of mirrors in the firstorientation state will vary from one spatial pattern to the next. Thus,the cumulative intensity of light portions reflected onto the sensingpath 115 and arriving at the light sensing device will vary as afunction time. Note that the term “cumulative” is meant to suggest asummation over the light portions arriving at the light sensing deviceat any given time. As will be described below, that summation may beimplemented, at least in part, optically (e.g., by means of a lensand/or mirror that concentrates or focuses the light portions onto aconcentrated area as described above).

System realization 200 may include any subset of the features,embodiments and elements discussed above with respect to system 100. Forexample, system realization 200 may include the optical subsystem 105 tooperate on the incoming light L before it arrives at the mirrors 110M,e.g., as shown in FIG. 3B.

In some embodiments, system realization 200 may include the opticalsubsystem 117 along the sensing path as shown in FIG. 4. The opticalsubsystem 117 receives the light portions reflected onto the sensingpath 115 and directs (e.g., focuses or concentrates) the received lightportions onto a light sensing surface of the light sensing device 130.In one embodiment, the optical subsystem 117 may include a lens 117L,e.g., as shown in FIG. 5A.

In some embodiments, the optical subsystem 117 may include one or moremirrors, e.g., a mirror 117M as shown in FIG. 5B. Thus, the sensing pathmay be a bent path having more than one segment. FIG. 5B also shows onepossible embodiment of optical subsystem 105, as a lens 105L.

In some embodiments, there may be one or more optical elementsintervening between the optical subsystem 105 and the mirrors 110M. Forexample, as shown in FIG. 5C, a TIR prism pair 107 may be positionedbetween the optical subsystem 105 and the mirrors 110M. (TIR is anacronym for “total internal reflection”.) Light from optical subsystem105 is transmitted through the TIR prism pair and then interacts withthe mirrors 110M. After having interacted with the mirrors 110M, lightportions from mirrors in the first orientation state are reflected by asecond prism of the pair onto the sensing path 115. Light portions frommirrors in the second orientation state may be reflected away from thesensing path.

Calibration Patterns Interspersed Among Measurement Patterns

In one set of embodiments, a system 600 for operating on a stream oflight from an environment may be configured as shown in FIG. 6A.

The system 600 may include the light modulation unit 110, the lightsensing device 130 and the processing unit 150 as described above.(Furthermore, any subset of the features, embodiments and elementsdiscussed above with respect to system 100 and system realization 200may be incorporated into system 600.)

In system 600, the mechanism 140 for measuring background light level isrealized by including calibration patterns as well as measurementpatterns in the series of spatial patterns supplied to the lightmodulation unit 110. Thus, the sequence of samples {I_(MLS)(k)}generated by the light sensing device 130 will include samples (e.g.,compressive sensing measurements) corresponding to the measurementpatterns and samples corresponding to the calibration patterns. Thecalibration patterns are designed so that the corresponding samples willrepresent variation in the background light level. There are a widevariety of ways to satisfy this design constraint.

In some embodiments, each of the calibration patterns specifies that allthe light modulating elements (of the light modulation unit 110) shouldassume the maximum transfer state. The maximum transfer state of a lightmodulating element is the state in which the ratio of the intensity ofthe light portion outputted from the element onto the light sensing path115 to the intensity of the light portion incident upon the element ismaximized, within the range of control supported by the element. (Forexample, for an element having controllable transmittance, such as anLCD shutter, the maximum transfer state is the state of maximumtransparency. As another example, for a reflective element such as amicromirror in a DMD, the maximal transfer state is the orientationstate that reflects the received light portion onto the sensing path115.) Thus, when one of the calibration patterns is being asserted onthe light modulation unit 110, the modulated light stream MLS mostclosely approximates the incident light stream L (among all possiblespatial patterns that could be applied by the light modulation unit110). The samples captured by the light sensing device 130 when thecalibration patterns are being applied correspond to (are representativeof) the intensity of the incident light stream L (or a fixed fraction ofthat intensity).

In some embodiments, each of the calibration patterns specifies that allthe light modulating elements should assume the p transfer state, wherep is a fixed fraction greater than zero and less than or equal to one.The p transfer state of a light modulating element is the state in whichthe ratio of the intensity of the light portion leaving the element,relative to the intensity of the light portion entering the element, isequal to p. (For example, the 0.5 transfer state might be realized bysetting a controllable-transmittance element to its state of 50%transmittance, or, by controlling a digital micromirror to spend half ofa sampling period is the first orientation state and the other half ofthe sampling period in the second orientation state.) As a result, thesamples captured when the calibration patterns are being appliedcorrespond to (are representative of) 50% of the intensity of theincident light stream L.

In another embodiment, each of the calibration patterns is identical toa given maximum-resolution checkerboard pattern, where half the lightmodulating elements are ON (maximal transfer) and the remaining half areOFF (minimal transfer). (A maximum—resolution checkerboard pattern is apattern where the elements immediately north, south, east and west ofany given interior element take the state that is opposite to the stateof the given interior element.) Consequently, when any one of thecalibration patterns is being asserted, the modulated light stream MLSrepresents approximately 50% of the power in the incident light streamL. The samples captured by the light sensing device 130 when thecalibration patterns are being asserted will correspond to samples ofapproximately ½ of the intensity (or power) of the light stream L.

In another embodiment, the checkerboard pattern may be oflower-resolution, e.g., a checkerboard pattern whose fundamental unitsare L×L squares with L>1, e.g., 2×2 squares or 4×4 squares.

In yet another embodiment, the calibration patterns are not allidentical. For example, the calibration patterns may alternate between afirst checkerboard pattern and a second checkerboard pattern, where thefirst and second checkerboard patterns are complementary to each other:one takes the ON state where the other takes the OFF state, and viceversa.

As variously described above, the light modulation unit 110 isconfigured to modulate the light stream L with a series of spatialpatterns in order to produce a modulated light stream MLS. (See FIG.6A.) The light modulation unit includes a plurality of light modulatingelements configured to modulate corresponding portions of the lightstream. Each of the spatial patterns specifies an amount of modulation(or scalar multiplier) for each of the light modulating elements. Inother words, each spatial pattern P specifies a collection of controlvalues for the light modulating elements, one control value per element.A control value controls the amount of modulation applied by thecorresponding light modulating element. The series of spatial patternsincludes measurement patterns and calibration patterns, e.g., asvariously described above.

The light sensing device 130 is configured to receive the modulatedlight stream MLS and to generate a sequence of samples {I_(MLS)(k)}representing intensity of the modulated light stream as a function oftime, e.g., as variously described above.

The processing unit 150 is configured to receive the sequence of samplesgenerated by the light sensing device 130 and to separate the sequenceinto a first subsequence of samples that correspond to the measurementpatterns and a second subsequence of samples that correspond to thecalibration patterns. The samples of the first subsequence may beinterpreted as compressive sensing (CS) measurements. The secondsubsequence of samples represents variation in background light level.

The processing unit 150 may be configured to store the first subsequenceand second subsequence of samples in a memory. The first subsequence andsecond subsequence are usable to construct an image or a video sequence,and/or, to perform an inference task as variously described above.

The samples of the second subsequence may be used to compensate thesamples of the first subsequence for the variation in the backgroundlight level, e.g., as variously described above. The compensated samplesare usable to construct the image or video sequence, and/or, to performan inference task as variously described above. The compensation may beperformed by system 600 or by some other system.

Regarding the measurement patterns, a wide variety of realizations arepossible. In some embodiments, the measurement patterns may belong to ameasurement vector set that is incoherent with respect to a sparsityvector set that will be used to construct the image or video sequence.

Referring now to FIG. 6B, system 600 may also include the control unit120 as described above. The control unit supplies the series of spatialpatterns to the light modulation unit 110. The control unit may include(or insert) the calibration patterns in the series of spatial patternsaccording to any of a variety of methods. In one embodiment, thecalibration patterns may be included in series of patterns in a periodicfashion, i.e., one out of every N_(P) patterns in the series may be acalibration pattern and the remaining N_(P)−1 patterns may bemeasurement patterns. In another embodiment, the calibration patternsmay be included in the series of patterns with a variable rate thatdepends on rate of change of the background light level or on thevariance of a set of most recent samples of the background light level.For example, the processing unit 150 may analyze the samples of thebackground light level to determine the rate of change r_(BGL) of thebackground light level. The rate at which the calibration patterns areincluded in the series of patterns may increase as r_(BGL) increases.

In some embodiments of system 600, the light modulation unit 110 may berealized by the mirrors 110M, as shown in FIG. 6C. In these embodiments,the control unit 120 is configured to drive the orientation states ofthe mirrors through the series of patterns. As described above, theseries of patterns includes both measurement patterns and calibrationpatterns. Thus, the sequence of samples generated by the light sensingdevice 130 will include samples corresponding to the measurementpatterns and samples corresponding to the calibration patterns. Thesamples corresponding to the calibration patterns (i.e., the samples ofthe second subsequence) represent variation of the background lightlevel.

In one embodiment, each of the calibration patterns specifies that allthe mirrors should be in the first orientation state that reflects lightonto the sensing path 115.

In another embodiment, each of the calibration patterns is identical toa checkerboard pattern.

In one set of embodiments, a method 700 for operating on a stream oflight may involve the actions shown in FIG. 7A. In one embodiment, themethod 700 may be performed using the system 600 described above.

Action 710 includes modulating the light stream with a series of spatialpatterns in order to produce a modulated light stream, e.g., asvariously described above. The series of spatial patterns includesmeasurement patterns and calibration patterns, e.g., as variouslydescribed above.

Action 715 includes generating a sequence of samples representingintensity of the modulated light stream as a function of time, e.g., asvariously described above.

Action 720 includes separating the sequence of samples into a firstsubsequence of samples that correspond to the measurement patterns and asecond subsequence of samples that correspond to the calibrationpatterns. The second subsequence of samples represents variation inbackground light level.

In one embodiment, the method 700 may also include storing the firstsubsequence of samples and the second subsequence of samples in amemory. The first subsequence and the second subsequence of samples areusable to construct an image or a video sequence, and/or, to perform aninference task as variously described above.

In one embodiment, the method 700 may also include compensating thefirst subsequence of samples for the variation in background light levelusing the second subsequence of samples, e.g., as variously describedabove.

In one embodiment, the method 700 may also include constructing theimage or the video sequence based on the compensated samples, e.g., asvariously described above.

In one embodiment, the method 700 may also include transmitting thecompensated samples to a receiver through a communication channel. Thereceiver may be configured to construct the image or the video sequencebased on the compensated samples, and/or, to perform the inference taskas described above.

In some embodiments, a non-transitory computer-readable memory mediummay be configured so that it stores program instructions, where theprogram instructions, when executed by a computer system, cause thecomputer system to: (1) supply a series of spatial patterns to a lightmodulation unit, where the light modulation unit is configured tomodulate a received light stream with the series of spatial patterns toobtain a modulated light stream, where the series of spatial patternsincludes first patterns and calibration patterns (e.g., as variouslydescribed above); (2) receive a sequence of samples from a first lightsensing device (e.g., the light sensing device as variously describedabove), where the sequence of samples represents intensity of themodulated light stream as a function of time; and (3) separate thesequence of samples into a first subsequence of samples corresponding tothe first patterns and a second subsequence of samples corresponding tothe calibration patterns, where the second subsequence of samplesrepresents variation of background light level in the environment.

In some embodiments, a method 750 (e.g., a method for operating onlight) may involve the actions shown in FIG. 7B. In one embodiment, themethod may be performed using the system 600 described above.

Action 760 includes controlling a plurality of mirrors. The mirrors areconfigured to reflect corresponding portions of a received beam oflight, where each of the mirrors is configured to controllably switchbetween two orientation states. The action of controlling the mirrorsincludes driving the mirrors through a series of spatial patterns, whereeach of the spatial patterns specifies an orientation state for each ofthe mirrors. The series of spatial patterns includes measurementpatterns and calibration patterns, e.g., as variously described above.

Action 770 includes receiving a sequence of samples from a light sensingdevice (e.g., the light sensing device as described above). The sequenceof samples represents cumulative intensity of light portions reflectedby mirrors in a first of the two orientation states as a function oftime.

Action 780 includes separating the sequence of samples into a firstsubsequence of samples corresponding to the measurement patterns and asecond subsequence of samples corresponding to the calibration patterns.The second subsequence of samples represents variation of backgroundlight level in the environment.

Various additional embodiments as described in the following numberedparagraphs.

1.1 A system for operating on a stream of light, the system comprising:

a light modulation unit configured to modulate the light stream with aseries of spatial patterns in order to produce a modulated light stream,wherein the light modulation unit includes a plurality of lightmodulating elements configured to modulate corresponding portions of thelight stream, wherein each of the spatial patterns specifies an amountof modulation for each of the light modulating elements, wherein theseries of spatial patterns includes first patterns and calibrationpatterns;

a light sensing device configured to receive the modulated light streamand to generate a sequence of samples representing intensity of themodulated light stream as a function of time; and

a processing unit configured to separate the sequence of samples into afirst subsequence of samples that correspond to the first patterns and asecond subsequence of samples that correspond to the calibrationpatterns, wherein the second subsequence of samples represents variationin background light level.

1.2 The system of paragraph 1.1, further comprising a first opticalsubsystem configured to receive the light stream and to focus thereceived light stream onto the light modulation unit.

1.3 The system of paragraph 1.1, wherein the processing unit isconfigured to store the first subsequence and second subsequence ofsamples in a memory, wherein the first subsequence and secondsubsequence are usable to construct an image or a video sequence.

1.4 The system of paragraph 1.1, wherein the samples of the secondsubsequence are used to compensate the samples of the first subsequencefor the variation in the background light level, wherein the compensatedsamples are usable to construct an image or a video sequence.

1.5 The system of paragraph 1.4, wherein the processing unit isconfigured to perform said compensation of the samples of the firstsubsequence using the samples of the second subsequence.

1.6 The system of paragraph 1.4, wherein the processing unit isconfigured to transmit the compensated samples to a receiver through acommunication channel, wherein the receiver is configured to constructthe image or the video sequence based on the compensated samples.

1.7 The system of paragraph 1.1, wherein the plurality of lightmodulating elements are mirrors whose orientations are independentlycontrollable.

1.8 The system of paragraph 1.1, wherein the plurality of lightmodulating elements are elements whose transmittances are independentlycontrollable.

1.9 The system of paragraph 1.1, further comprising a second opticalsubsystem configured to direct the modulated light stream onto a lightsensing surface of the light sensing device.

1.10 The system of paragraph 1.1, wherein the modulation amountsspecified by the spatial patterns are binary values.

1.11 The system of paragraph 1.1, wherein the light sensing deviceincludes one or more photodiodes.

1.12 The system of paragraph 1.1, wherein the plurality of lightmodulating elements are arranged in a two-dimensional array.

1.13 The system of paragraph 1.1, wherein the first patterns arepseudo-random patterns.

1.14. The system of paragraph 1.1, wherein the calibration patterns areincluded in the series of patterns in a periodic fashion.

1.15 A method for operating on a stream of light, the method comprising:

modulating the light stream with a series of spatial patterns in orderto produce a modulated light stream, wherein the series of spatialpatterns includes first patterns and calibration patterns;

generating a sequence of samples representing intensity of the modulatedlight stream as a function of time; and

separating the sequence of samples into a first subsequence of samplesthat correspond to the first patterns and a second subsequence ofsamples that correspond to the calibration patterns, wherein the secondsubsequence of samples represents variation in background light level.

1.16 The method of paragraph 1.15, further comprising storing the firstsubsequence of samples and the second subsequence of samples in amemory, wherein the first subsequence and the second subsequence ofsamples are usable to construct an image or a video sequence.

1.17 The method of paragraph 1.15, further comprising compensating thefirst subsequence of samples for the variation of background light levelusing the second subsequence of samples.

1.18. The method of paragraph 1.17, further comprising constructing animage or a video sequence based on the compensated samples.

1.19. The method of paragraph 1.17, further comprising transmitting thecompensated samples to a receiver through a communication channel.

1.20. The method of paragraph 1.15, wherein the first patterns arepseudo-random patterns.

1.21 The method of paragraph 1.15, wherein each of the calibrationpatterns is a constant pattern specifying that all the light modulatingelements assume a maximal transfer state.

1.22 A non-transitory computer-readable memory medium storing programinstructions, wherein the program instructions, when executed by acomputer system, cause the computer system to: supply a series ofspatial patterns to a light modulation unit, wherein the lightmodulation unit is configured to modulate a received light stream withthe series of spatial patterns to obtain a modulated light stream,wherein the series of spatial patterns includes first patterns andcalibration patterns; receive a sequence of samples from a first lightsensing device, wherein the sequence of samples represents intensity ofthe modulated light stream as a function of time; and separate thesequence of samples into a first subsequence of samples corresponding tothe first patterns and a second subsequence of samples corresponding tothe calibration patterns, wherein the second subsequence of samplesrepresents variation of background light level.

1.23 A system for operating on light from an environment, the systemcomprising:

a plurality of mirrors, wherein each of the mirrors is configured tocontrollably switch between two orientation states, wherein each of themirrors is configured to (a) reflect a corresponding portion of thelight onto a sensing path when the mirror is in a first of the twoorientation states and (b) reflect the corresponding portion of thelight away from the sensing path when the mirror is in a second of thetwo orientation states;

a control unit configured to drive the orientation states of the mirrorsthrough a series of spatial patterns, wherein each of the patterns ofthe series specifies an orientation state for each of the mirrors,wherein the series of patterns includes first patterns and calibrationpatterns;

a light sensing device configured to receive light portions reflectedonto the sensing path by mirrors in the first orientation state and togenerate a sequence of samples representing a cumulative intensity ofthe light portions reflected onto the sensing path as a function oftime; and

a processing unit configured to separate the sequence of samples into afirst subsequence of samples that correspond to the first patterns and asecond subsequence of samples that correspond to the calibrationpatterns, wherein the second subsequence of samples represents variationin background light level.

1.24. The system of paragraph 1.23, further comprising a first opticalsubsystem configured to receive the light from the environment and tofocus the received light onto the plurality of mirrors.

1.25. The system of paragraph 1.24, further comprising a TIR prism pairpositioned between the first optical subsystem and the plurality ofmirrors.

1.26. The system of paragraph 1.23, further comprising a second opticalsubsystem configured to receive the light portions reflected onto thesensing path by mirrors in the first orientation state, and to directthe received light portions onto a light sensing surface of the lightsensing device.

1.27 A method comprising:

controlling a plurality of mirrors, wherein the mirrors are configuredto reflect corresponding portions of a received beam of light, whereineach of the mirrors is configured to controllably switch between twoorientation states, wherein said controlling the mirrors includesdriving the mirrors through a series of spatial patterns, wherein eachof the spatial patterns specifies an orientation state for each of themirrors, wherein the series of spatial patterns includes first patternsand calibration patterns;

receiving a sequence of samples from a first light sensing device,wherein the sequence of samples represents cumulative intensity of lightportions reflected by mirrors in a first of the two orientation statesas a function of time;

separating the sequence of samples into a first subsequence of samplescorresponding to the first patterns and a second subsequence of samplescorresponding to the calibration patterns, wherein the secondsubsequence of samples represents variation of background light level inthe environment.

1.28 The method of paragraph 1.27, wherein each of the calibrationpatterns is a pattern specifying that all the mirrors be in the firstorientation state.

Sensing Light Reflected by Mirrors in Both Orientation States

In one set of embodiments, a system 800 for operating on incident lightfrom an environment may be configured as shown in FIG. 8. System 800includes the mirrors 110M, the control unit 120 and the light sensingdevice 130 as described above in connection with system realization 200,and also includes a second light sensing device 135 to be describedpresently. (Any subset of the features, embodiments and elementsdiscussed above with respect to system 100 and system realization 200may be incorporated into system 800.) While light sensing device 130 issensitive to the light portions reflected by mirrors in the firstorientation state, light sensing device 135 is sensitive to the lightportions reflected by mirrors in the second orientation state. Thus, thebackground light level (or the intensity of the input light L) may beinferred by appropriately combining the intensity signals (the outputs)of the two light sensing devices.

In system 800, an alternative sensing path 116 is provided, i.e.,alternative to the sensing path 115. The alternative path extends fromthe mirrors 110M to the light sensing device 135.

The mirrors 110M are configured to receive corresponding portions of theinput light L. Each of the mirrors is configured to controllably switchbetween two orientation states. Furthermore, each of the mirrors isconfigured to (a) reflect the corresponding portion of the light L ontothe sensing path 115 when the mirror is in the first orientation stateand (b) reflect the corresponding portion of the light L onto thealternative path 116 when the mirror is in the second orientation state.

The first light sensing device 130 is configured to receive the firstsubset of light portions that are reflected onto the sensing path 115 bymirrors in the first orientation state and to generate a first sequenceof samples {I₁(k)} representing a cumulative intensity of the firstsubset of light portions as a function of time. (The first subsetcontinuously changes as the control unit 120 drives the mirrors 110Mthrough the series of spatial patterns.) The samples of the firstsequence {I₁(k)} comprise an encoded (e.g., compressed) representationof an image or video sequence.

The light sensing device 135 is configured to receive the second subsetof light portions that are reflected onto the alternative path 116 bymirrors in the second orientation state and to generate a secondsequence of samples {I₂(k)} representing a cumulative intensity of thesecond subset of light portions as a function of time. (As with thefirst subset, the second subset of light portions continuously changesas the control unit 120 drives the mirrors 110M through the series ofspatial patterns.) The samples of the second sequence {I₂(k)} representan encoded (e.g., compressed) representation of the same image or videosequence. The first sequence {I₁(k)} and second sequence {I₂(k)} may beinterpreted as complementary encodings of the same image (or videosequence) because the spatial patterns as seen by the light sensingdevice 130 are complementary to the spatial patterns as seen by thelight sensing device 135.

In some embodiments, the light sensing devices 130 and 135 may bedesigned to have the same (or similar) parameters, e.g., sensitivitiesto incident light, areas of light receiving surface, etc. Alternatively,the light sensing devices 130 and 135 may be different by design.

The first sequence {I₁(k)} and the second sequence {I₂(k)} may becombined to obtain a third sequence {I_(BGL)(k)} that representsvariation in the background light level. In some embodiments, thesamples of the third sequence {I_(BGL)(k)} may be used to compensate thesamples of the first sequence {I₁(k)} for the variation in backgroundlight level, e.g., as variously described above. The compensated samplesmay be used to construct an image or a video sequence, e.g., using anyof the various construction methods known in the field of compressivesensing. As noted above, the compensated samples may be used to performa task other than or in addition to image construction, e.g., to performdetection, classification, tracking, etc.

There are various ways of combining the first sequence {I₁(k)} and thesecond sequence {I₂(k)} to obtain the third sequence {I_(BGL)(k)}, e.g.,depending on the nature of physical quantity measured by the lightsensing devices 130 and 135. In one embodiment, the third sequence{I_(BGL)(k)} may be determined based on the expression:I _(BGL)(k)=I ₁(k)+I ₂(k),or more generally,I _(BGL)(k)=αI ₁(k)+βI ₂(k),where α and β are positive real constants. (The constants α and β may beused to compensate for any imbalance in the delivery of light powerbetween the sensing path and the alternative path and/or any imbalancein the sensitivities of the light sensing devices 130 and 135.) In someembodiments, I_(BGL)(k) may be determined based on a combination ofg(I₁(k)) and g(I₂(k)), where g(*) is a strictly increasing function(such as a logarithm function or the square-root function). Yet moregenerally, I_(BGL)(k) may be determined based on a functionalcombination of I₁(k) and I₂(k):I _(BGL)(k)=f(I ₁(k),I ₂(k)),where f(*,*) is a function of two real variables. The function f may beincreasing in each variable.

In one embodiment, the analog signals I₁(t) and I₂(t) generated by thelight sensing devices may be combined (e.g., added) in the analog domainto obtain an analog signal I_(BGL)(t) that represents variation in thebackground light level. The analog signal I_(BGL)(t) may then bedigitized to obtain the samples {I_(BGL)(k)}.

As shown in FIG. 9, system 800 may include the processing unit 150described above in connection with system 100 and system realization200. The processing unit 150 may be configured to perform the action ofcombining the first sequence {I₁(k)} and the second sequence {I₂(k)} ofsamples to obtain the third sequence {I_(BGL)(k)} of samples.Furthermore, the processing unit 150 may be configured to compensate thesamples of the first sequence {I₁(k)} for the variation in backgroundlight level using the samples of the third sequence {I_(BGL)(k)}, e.g.,as variously described above. In some embodiments, the action ofcompensating the samples {I₁(k)} may be performed by some other agent(e.g., digital logic) that resides in system 800 or outside of system800.

The first sequence {I₁(k)} and the third sequence {I_(BGL)(k)} may bestored in a memory that resides in system 800 or in some other system.The first sequence and third sequence may be used to construct an imageor a video sequence using any of the various construction methods knownto compressive sensing theory.

In some embodiments, the first sequence {I₁(k)} and the second sequence{I₂(k)} may be stored in a memory.

Referring again to FIG. 9, system 800 may also include the opticalsubsystem 105 as variously described above in connection with system 100and system realization 200.

As shown in FIG. 10, system 800 may include the optical subsystem 117 asvariously described above in connection with system 100 and systemrealization 200. The optical subsystem 117 is configured to receive thefirst subset of light portions that are reflected onto the sensing path115 by mirrors in the first orientation state, and to direct (e.g.,focus or concentrate) the first subset of light portions onto a lightsensing surface of the light sensing device 130. Furthermore, system 800may include an optical subsystem 118. The optical subsystem 118 isconfigured to receive the second subset of light portions that arereflected onto the alternative path 116 by mirrors in the secondorientation state, and to direct (e.g., focus or concentrate) the secondsubset of light portions onto a light sensing surface of the lightsensing device 135.

In some embodiments, the optical subsystems 117 and 118 may be designedto have the same (or similar) power attenuation.

In some embodiments, optical subsystems 117 and 118 are identical (ornearly identical) in construction. For example, the two subsystem mayhave matching sets of components that are spatially arranged in anidentical (or similar) fashion, e.g., according to a mirror-imagespatial symmetry.

Optical subsystems 117 and 118 may include components such as one ormore lenses and/or one or more mirrors. FIG. 11 shows optical subsystem117 as having a mirror 117M and a lens 117L, and optical subsystem 118as having a mirror 118M and a lens 118L.

In some embodiments, light sensing device 130 may include a plurality oflight sensing elements (e.g., photodiodes) arranged in an array (orotherwise distributed). The signals (e.g., photocurrents) from the lightsensing elements may be sampled by any of various means. The samplesfrom each light sensing element may be processed to construct acorresponding sub-image. Each sub-image corresponds to a respectivesub-region of the array of mirrors 110M. Furthermore, the signals (e.g.,photocurrents) from all, from a subset or from a number of subsets ofthe light sensing elements may be summed. The summed signal may besampled to obtain the signal I₁(k). The signal I₁(k) may be combinedwith the signal I₂(k) to infer the background light level as a functionof time (or the total amount of light in the scene being imaged as afunction of time) during the data acquisition process. In someembodiments, light sensing device 135 may also include a plurality oflight sensing elements (e.g., photodiodes) arranged in an array (orotherwise distributed). The signal I₂(k) may be similarly generatedbased on a sum of signals from all, from a subset or from a number ofsubsets of those light sensing elements.

In some embodiments, the light sensing device 135 includes (or is) asingle photodetector such as a photodiode.

In some embodiments, the light sensing device 135 is a spectrometer.

In some embodiments, the light sensing device 135 is configured to sensea different wavelength band than the light sensing device 130. Forexample, light sensing device 135 may include a photodiode covering adifferent wavelength band than the wavelength band covered by lightsensing device 130.

It is noted that the presence of more than one light sensing device insystem 800 allows system 800 to continue functioning as a compressiveimaging system if one of the light sensing devices should fail (orotherwise become unreliable). Thus, system 800 (e.g., processing unit150) may be configured to monitor the signals (e.g., sample sequencesI₁(k) and I₂(k) described above) generated by the light sensing devices,and detect any irregular pattern of behavior. If one of the lightsensing devices is exhibiting irregular behavior, the system 800 mayignore the signal generated by that light sensing device and use onlythe signal generated by the remaining (properly functioning) lightsensing device to acquire the compressive imaging measurements.

In some embodiments, a method 1200 for operating on light may includethe actions shown in FIG. 12. In one embodiment, method 1200 may beperformed using system 800.

Action 1210 includes controlling a plurality of mirrors (e.g., themirrors 110M as described above). The mirrors are configured to reflectcorresponding portions of a received beam of light. Each of the mirrorsis configured to controllably switch between two orientation states. Theoperation of controlling the mirrors includes driving the mirrorsthrough a series of spatial patterns, where each of the spatial patternsspecifies an orientation state for each of the mirrors. The spatialpatterns of the series may be as variously described above. For example,in one embodiment, the spatial patterns may be random patterns orpseudo-random patterns.

Action 1220 includes receiving a first sequence {I₁(k)} of samples froma first light sensing device. The first sequence of samples representscumulative intensity of light portions reflected by mirrors in a firstof the two orientation states as a function of time.

Action 1230 includes receiving a second sequence {I₂(k)} of samples froma second light sensing device. The second sequence of samples representscumulative intensity of light portions reflected by mirrors in a secondof the two orientation states as a function of time.

Action 1240 includes combining the first sequence and the secondsequence of samples to obtain a third sequence {I_(BGL)(k)} of samples,e.g., as variously described above. The third sequence of samplesrepresents variation of the background light level.

In one embodiment, method 1200 also includes storing the first sequence{I₁(k)} and the third sequence {I_(BGL)(k)} of samples in a memory. Thefirst sequence and the third sequence of samples may be used toconstruct an image or video sequence.

In one embodiment, the method 1200 also includes compensating the firstsequence {I₁(k)} of samples for the variation of background light levelusing the third sequence {I_(BGL)(k)} of samples, e.g., as variouslydescribed above. In one embodiment, the method 1200 may also includeconstructing an image or video sequence based on the compensatedsamples.

The method 1200 may also include transmitting the compensated samples toa receiver through a communication channel. The receiver may beconfigured to construct the image or the video sequence based on thecompensated samples.

In some embodiments, a non-transitory computer-readable memory mediummay be configured so that it stores program instructions, wherein theprogram instructions, when executed by a computer system, cause thecomputer system to: (a) control a plurality of mirrors, wherein themirrors are configured to reflect corresponding portions of a receivedbeam of light, wherein each of the mirrors is configured to controllablyswitch between two orientation states, wherein said controlling themirrors includes driving the mirrors through a series of spatialpatterns, wherein each of the spatial patterns specifies an orientationstate for each of the mirrors; (b) receive a first sequence of samplesfrom a first light sensing device, wherein the first sequence of samplesrepresents cumulative intensity of light portions reflected by mirrorsin a first of the two orientation states as a function of time; (c)receive a second sequence of samples from a second light sensing device,wherein the second sequence of samples represents cumulative intensityof light portions reflected by mirrors in a second of the twoorientation states as a function of time; and (d) combine the firstsequence and the second sequence of samples to obtain a third sequenceof samples that represents variation of background light level.

Note that the technical principle of inferring background light level bycombining orientation-specific intensity signals applies more generallyto systems where the mirrors 110M have two or more orientation statesper mirror. For a system of mirrors where each mirror has n orientationstates, n≧2, the system may include n corresponding light sensingdevices, each light sensing device configured to capture light frommirrors in a corresponding one of the n orientation states.

Various additional embodiments are described in the following numberedparagraphs.

2.1. A system for operating on light, the system comprising:

a plurality of mirrors configured to receive corresponding portions ofthe light, wherein each of the mirrors is configured to controllablyswitch between two orientation states, wherein each of the mirrors isconfigured to (a) reflect the corresponding portion of the light onto asensing path when the mirror is in a first of the two orientation statesand (b) reflect the corresponding portion of the light onto analternative path when the mirror is in a second of the two orientationstates;

a control unit configured to drive the orientation states of the mirrorsthrough a series of patterns, wherein each of the patterns of the seriesspecifies an orientation state for each of the mirrors;

a first light sensing device configured to receive first light portionsreflected onto the sensing path by mirrors in the first orientationstate and to generate a first sequence of samples representing acumulative intensity of the first light portions as a function of time;

a second light sensing device configured to receive second lightportions reflected onto the alternative path by mirrors in the secondorientation state and to generate a second sequence of samplesrepresenting a cumulative intensity of the second light portions as afunction of time.

2.2. The system of paragraph 2.1, further comprising a memory thatstores the first sequence of samples and a third sequence of samples,wherein the third sequence of samples is a combination of the firstsequence and the second sequence of samples and represents variation inbackground light level.

2.3. The system of paragraph 2.2, wherein the samples of the thirdsequence are used to compensate the samples of the first sequence forthe variation in background light level, wherein the compensated samplesare usable to construct an image or a video sequence.

2.4. The system of paragraph 2.1, further comprising a processing unitconfigured to combine the first sequence and the second sequence ofsamples to obtain a third sequence of samples that represents variationin background light level, wherein the samples of the third sequence areused to compensate the samples of the first sequence for the variationin background light level, wherein the compensated samples are usable toconstruct an image or a video sequence.

2.5. The system of paragraph 2.4, wherein the processing unit isconfigured to perform said compensation of the samples of the firstsequence using the samples of the third sequence.

2.6. The system of paragraph 2.1, further comprising an optical inputpath configured to receive the light and to focus the received lightonto the plurality of mirrors.

2.7. The system of paragraph 2.1, further comprising a first opticalsubsystem configured to receive the first light portions reflected ontothe sensing path and to direct the first light portions onto a lightsensing surface of the first light sensing device.

2.8. The system of paragraph 2.7, wherein the first optical systemincludes a lens.

2.9. The system of paragraph 2.1, further comprising a second opticalsubsystem configured to receive the second light portions reflected ontothe alternative path and to direct the second light portions onto alight sensing surface of the second light sensing device.

2.10. The system of paragraph 2.9, wherein the second optical systemincludes a lens.

2.11. The system of paragraph 2.1, wherein the first light sensingdevice includes one or more light sensing elements.

2.12. The system of paragraph 2.1, wherein the second light sensingdevice includes one or more light sensing elements.

2.13. The system of paragraph 2.1, wherein the plurality of mirrors arearranged in an array.

2.14. The system of paragraph 2.1, wherein the patterns arepseudo-random patterns.

Measure Light that is Separated from the Optical Train Prior to theLight Modulation Unit.

In one set of embodiments, a system 1300 for operating on light may beconfigured as shown in FIG. 13A. System 1300 includes the lightmodulation unit 110 and the light sensing device 130 as described abovein connection with system 100, and also includes an optical subsystem1310 and a light sensing device 1320. (Furthermore, any subset of thefeatures, embodiments and elements discussed above with respect tosystem 100 and system realization 200 may be incorporated into system1300.)

The optical subsystem 1310 may be configured to receive the light streamL and to separate the light stream L into a first light stream B₁ and asecond light stream B₂. The first light stream B₁ is supplied to thelight modulation unit 110. The second light stream B₂ is supplied to thelight sensing device 1320. The output of the light sensing device 1320represents (or may be used to infer) variation in background lightlevel.

The light modulation unit 110 is configured to modulate (as describedvariously above) the first light stream B₁ with a series of spatialpatterns in order to produce a modulated light stream MLS. The lightsensing device 130 is configured to receive the modulated light streamMLS and to generate a sequence of samples {I_(MLS)(k)} representingintensity of the modulated light stream as a function of time. Index kis a time index. The samples of the sequence {I_(MLS)(k)} comprise anencoded (e.g., compressed) representation of an image or video sequence.

The light sensing device 1320 is configured to generate a sequence{I_(BGL)(k)} of samples of intensity of the second light stream B₂ as afunction of time. As noted above, the samples {I_(BGL)(k)} correspond to(or may be used to infer) variation in background light level, i.e.,background light level as a function of time. The light sensing device1320 may include one or more light sensing elements (e.g., photodiodes).

In one embodiment, the light sensing device 1320 may be fabricated usinga different photodiode material technology than is used in light sensingdevice 130.

The sequence {I_(MLS)(k)} and the sequence {I_(BGL)(k)} may be stored ina memory, e.g., a memory resident in system 1300 or elsewhere. Thesequence {I_(MLS)(k)} and the sequence {I_(BGL)(k)} may be used toconstruct an image or a video sequence. In particular, the sequence{I_(BGL)(k)} may be used to compensate the sequence {I_(MLS)(k)} for thevariation in the background light level, e.g., as described above inconnection with system 100. The compensated samples {I_(CMPN)(k)} maythen be used to construct the image or the video sequence, and/or, toperform an inference task (e.g., identification, classification,tracking, etc.) as variously described above.

System 1300 may be contrasted with the system 1350 of FIG. 13B. System1350 includes an optical subsystem 1360 that separates the incidentlight stream L into a first light stream Y₁ and a second light streamY₂, but does not include a device to measure/sense the second lightstream Y₂. In some embodiments, the optical subsystem 1360 may beselected based on design constraints that do not include its ability togenerate the second light stream Y₂. The generation of the second lightstream Y₂ may be a by-product of the selected design. For example, inFIG. 13C, the TIR prism pair 107 may be selected for its ability toreflect the light that leaves the mirrors 110M in the first orientationstate onto the path leading to the subsystem 117. However, the TIR prismpair 107 also generates the reflection Y₂. In some embodiments, system1300 may be a modification of system 1350, where a light sensing deviceis added to sense the intensity of the light stream Y₂, and therebyproduce an indication of the background light level as function of time.That modification may be interpreted as an improvement over system 1350because it uses the light stream Y₂ that would have been wasted insystem 1350. However, it is noted that other embodiments arecontemplated where system 1300 is not simply a modification of a systemsuch as system 1350. For example, in some embodiments, the opticalsubsystem 1310 of system 1300 may be selected based on designconstraints that include the ability to generate a second light streamB₂ of a certain kind or character or intensity.

As shown in FIG. 14, system 1300 may include the control unit 120 asdescribed above in connection with system 100. The control unit 120supplies the spatial patterns to the light modulation unit 110.

In some embodiments of system 1300, the light modulation unit 110 may berealized by the above-described mirrors 110M, e.g., as shown in FIG. 15.Each of the mirrors 110M is configured to receive a correspondingportion of the first light stream B₁, and is configured to controllablyswitch between two orientation states. Furthermore, each of the mirrorsis configured to: (a) reflect the corresponding portion of the firstlight stream onto the sensing path 115 when the mirror is in a first ofthe two orientation states and (b) reflect the corresponding portion ofthe first light stream away from the sensing path when the mirror is ina second of the two orientation states. The control unit 120 isconfigured to drive the orientation states of the mirrors through aseries of spatial patterns, where each of the patterns of the seriesspecifies an orientation state for each of the mirrors. The lightsensing device 130 is configured to receive the light portions reflectedonto the sensing path 115 by mirrors in the first orientation state andto generate a sequence {I_(MLS)(k)} of samples representing a cumulativeintensity of the light portions reflected onto the sensing path as afunction of time. Index k is a time index. The samples of the sequence{I_(MLS)(k)} may be interpreted as compressive sensing measurements.

Referring now to FIG. 16A, system 1300 may include the processing unit150 as described above in connection with system 100. The processingunit may be configured to compensate the samples of the sequence{I_(MLS)(k)} for the variation in the background light level using thesamples of the sequence {I_(BGL)(k)}. In one embodiment, the processingunit 150 may also be configured to generate an image or video sequenceusing the compensated samples, e.g., using any of the methods known incompressive sensing theory.

As described above, optical subsystem 1310 separates the light stream Linto a first light stream B₁ that is supplied to the light modulationunit 110 and a second light stream B₂ that is supplied to the lightsensing device 1320. After leaving the optical subsystem 1310, the firstlight stream and the second light stream may be optically isolated sothat light from the first light stream has minimal ability to arrive atlight sensing device 1320, and light from the second light stream hasminimal ability to interact with the light modulation unit 110.

In some embodiments, the first light stream B₁ may be a stream that istransmitted through the optical subsystem 1310, and the second lightstream B₂ may be a stream that is reflected from the first opticalsubsystem 1310.

In some embodiments, the optical subsystem 1310 is configured so thatthe first light stream B₁ is significantly more powerful that the secondlight stream B₂. In other words, most of the power of the light stream Lpasses into the first light stream. In some embodiments, the power ofthe second light stream is a small fraction of the power in the lightstream L, e.g., less than 20%, or less than 15% or less than 5% or lessthan 1% of the power in the light stream L.

There are a wide variety of ways the optical subsystem 1310 may berealized in hardware. For example, the optical subsystem 1310 mayinclude one or more following: a TIR prism pair, a spectral filter (suchas a grating, a thin film stack, a hologram or some other type of otherspectral filter), a beam splitter (e.g., in the form of a cube, plate,or pellicle), a thin film, a partially-mirrored surface, a polarizingbeam splitter, a fiber optic device, a waveguide.

FIGS. 16B and 16C show embodiments of system 1300 where the opticalsubsystem 1310 is embodied as a beam splitter cube 1310B. In FIG. 16Bthe light modulation unit 110 is realized by the mirrors 110M asvariously described above.

FIG. 16D shows an embodiment of system 1300 where the optical subsystem1310 is realized by a TIR prism pair 1310T.

In some embodiments, the optical subsystem 1310 may be configured sothat the second light stream B₂ is spectrally similar to the first lightstream B₁ (or to the received light stream L). In one embodiment, thisfeature may be achieved by virtue of the optical subsystem 1310 having apartially-reflective surface whose reflectance function andtransmittance function are flat over a broad range of wavelengths. Thebackground light measurements {I_(BGL)(k)} captured by the light sensingdevice 1320 may be scaled using knowledge of the relative magnitudes ofthe reflectance and transmittance functions in order to obtainbackground light values that are more appropriate to the first lightstream B₁, and thus, more appropriate for compensating the samples{I_(MLS)(k)} from the light sensing device 130.

In some embodiments, the optical subsystem 1310 may include a partiallyreflective surface (or some other mechanism for separating light) whosereflectance function and transmittance function are arbitrary functions,subject to fundamental physical constraints such as conservation ofenergy. Thus, the first light stream and the second light stream may notnecessarily be spectrally similar. However, the reflectance function andthe transmittance function may be known by calibration, and may beaccounted for when estimating the background light level. For example,in one embodiment, the light sensing device 130 and the light sensingdevice 1320 may be multi-spectral or hyper-spectral sensors. Thus, thelight sensing device 1310 may capture at each sampling time a set ofintensities {B₂(f_(i))} for respective frequencies {f_(i)} Similarly,the light sensing device 130 may capture at each sampling time a set ofintensities {I_(MLS)(f_(i))} for the respective frequencies {f_(i)}. Theset of intensities {B₂(f_(i))} may be adjusted based on knowledge of thereflectance function R(f) and the transmittance function T(f), to obtaina background light value for each frequency f_(i). For example, the set{B₂(f_(i))} may be adjusted according to the relationI _(BGL)(f _(i))=T(f _(i))/R(f _(i)))*B ₂(f _(i)).The background light value I_(BGL)(f_(i)) may be used to compensate theintensity sample I_(MLS)(f_(i)) captured by light sensing device 130 atthe same frequency f_(i).

More generally, regardless of the exact physical mechanism used toseparate the light stream L into first and second streams, the spectraltransfer function T₁(f) between the light stream L and the first lightstream B₁, and the spectral transfer function T₂(f) between the lightstream L and the second light stream B₂, may be determined bycalibration, and may be used to adjust the background light measurementscaptured by the light sensing device 1320 in order to obtain backgroundlight values more appropriate for the first light stream B₁, and thus,more appropriate for compensating the measurements {I_(MLS)(k)} from thelight sensing device 130.

In some embodiments, the optical subsystem 1310 may be configured sothat the second light stream B₂ is confined to a subset (or subrange) ofthe wavelengths present (or nominally present) in the received lightstream L. In some embodiment, this feature may be achieved by virtue ofthe optical subsystem 1310 having a partially-reflective surface whosereflectance function is supported only on the desired subset ofwavelengths. In one embodiment, this feature may be achieved by virtueof the optical subsystem 1310 including a grating or a spectral filter.

In some embodiments, the optical system 1310 is configured to separatethe received light stream based on polarization (e.g., using apolarizing beamsplitter cube) so that the first light stream B₁ and thesecond light stream B₂ have opposite polarization states. Even thoughthe first and second streams have opposite polarization states, they canstill be representative of the total light level (or the backgroundlight level) as a function of time. For example, the second lightstream, although having a polarization opposite to that of the firstlight stream, can still be representative of the total light level inthe first light stream or in the incident light stream L.

In some embodiments, the optical system 1310 is configured so that thesecond light stream B₂ has the same polarization state as the firstlight stream B₁.

In some embodiments, the second light stream may be a representativeportion (e.g., spatial portion) of the received light stream L takenfrom inside or near the field of view used by the set of mirrors 110M.

As shown in FIG. 16E, system 1300 may also include an optical subsystem1315 configured to receive the second light stream B₂, and to direct(e.g., focus or concentrate) the second light stream into an area (e.g.,a small area) on a light sensing surface of the light sensing device1320 (e.g., similar to the directing action of optical subsystem 117described above). The optical subsystem 1315 may include any set of oneor more optical components to achieve the directing function. Forexample, in one embodiment, optical subsystem 1315 may include one ormore lenses and/or one or more mirrors.

FIG. 16E also shows optical subsystem 117, which is configured toreceive the modulated light stream and to direct (e.g., focus orconcentrate) the modulated light stream onto a light sensing surface ofthe light sensing device 130, e.g., as variously described above. Insome embodiments, optical subsystems 117 and 1320 may be configuredsimilarly or identically. However, in other embodiments, they areconfigured differently, or even radically differently.

FIG. 16F shows one embodiment of the system shown in FIG. 16E, where thelight modulation unit 110 is realized by the mirrors 110M, e.g., asvariously described above.

In one set of embodiments, a method 1700 for operating on light (e.g.,light received from an environment) may involve the actions shown inFIG. 17A.

Action 1710 includes separating the light into a first light stream anda second light stream, e.g., as variously described above.

Action 1715 includes modulating the first light stream with a series ofspatial patterns in order to produce a modulated light stream, e.g., asvariously described above.

Action 1720 includes generating a first sequence of samples of intensityof the modulated light stream as a function of time, e.g., as variouslydescribed above.

Action 1725 includes generating a second sequence of samples ofintensity of the second light stream as a function of time, where thesecond sequence of samples corresponds to (or represents or indicates)variation in background light level (e.g., the background light level inthe environment).

In some embodiments, the method 1700 may also include storing the firstsequence and the second sequence of samples in a memory. The firstsequence and the second sequence of samples are usable to construct animage or a video sequence, and/or, to perform an inference task asvariously described above.

In some embodiments, the method 1700 may also include compensating thefirst sequence of samples for the variation of the background lightlevel using the second sequence of samples, e.g., as variously describedabove.

In some embodiments, the method 1700 may also include constructing animage or video sequence based on the compensated samples, and/or,performing an inference task, e.g., as variously described above.

In some embodiments, the method 1700 may also include transmitting thecompensated samples to a receiver through a communication channel, e.g.,as variously described above. (The receiver may be configured toconstruct the image or video sequence based on the compensated samples,and/or, to perform the inference task as variously described above.)

In one set of embodiments, a non-transitory computer-readable memorymedium may be configured so that it stores program instructions, wherethe program instructions, when executed by a computer system, cause thecomputer system to: (1) supply a series of spatial patterns to a lightmodulation unit (e.g., a light modulation unit as variously describedabove), where the light modulation unit is configured to modulate afirst light stream with the series of spatial patterns to obtain amodulated light stream, where the first light stream is produced by anoptical subsystem (or device) that separates an input light stream intothe first light stream and a second light stream (e.g., as variouslydescribed above); (2) receive a first sequence of samples from a firstlight sensing device (e.g., from the light sensing device 130 asvariously described above), where the first sequence of samplesrepresents intensity of the modulated light stream as a function oftime; (3) receive a second sequence of samples from a second lightsensing device (e.g., from the light sensing device 1320 as variouslydescribed above), where the samples of the second sequence are samplesof intensity of the second light stream as a function of time, where thesecond sequence of samples represents variation of background lightlevel; and (4) compensate the first sequence of samples for thevariation of the background light level using the second sequence ofsamples in order to obtain a compensated sequence of samples (e.g., asvariously described above), where the compensated samples are usable toconstruct an image or a video sequence, and/or, to perform an inferencetask as variously described above.

In one set of embodiments, a method 1750 for operating on light mayinvolve the actions shown in FIG. 17B.

Action 1760 includes separating input light L (received light) into afirst beam and a second beam, e.g., as variously described above.

Action 1765 includes controlling a plurality of mirrors, e.g., asvariously described above. The mirrors are configured to reflectcorresponding portions of the first beam. Each of the mirrors isconfigured to controllably switch between two orientation states. Theaction of controlling the mirrors includes driving the mirrors through aseries of spatial patterns, where each of the spatial patterns specifiesan orientation state for each of the mirrors.

Action 1770 includes receiving a first sequence of samples {I₁(k)} froma first light sensing device, where the first sequence of samples{I₁(k)} represents cumulative intensity of light portions reflected bymirrors in a first of the two orientation states as a function of time.The sample I₁(k) represents the cumulative intensity at a particulartime (or time interval) corresponding to time index value k.

Action 1775 includes receiving a second sequence of samples {I_(BGL)(k)}from a second light sensing device, where the samples of the secondsequence are samples of intensity of the second beam as a function oftime. The second sequence of samples represents variation of backgroundlight level.

In some embodiments, a non-transitory computer-readable memory medium isconfigured so that it stores program instructions, where the programinstructions, when executed by a computer system, cause the computersystem to: (1) control a plurality of mirrors, where the mirrors areconfigured to reflect corresponding portions of a first beam, where thefirst beam is produced by an optical subsystem that separates an inputlight beam into the first beam and a second beam, where each of themirrors is configured to controllably switch between two orientationstates, where the action of controlling the mirrors includes driving themirrors through a series of spatial patterns, where each of the spatialpatterns specifies an orientation state for each of the mirrors; (2)receive a first sequence of samples from a first light sensing device,where the first sequence of samples represents cumulative intensity oflight portions reflected by mirrors in a first of the two orientationstates as a function of time; and (3) receive a second sequence ofsamples from a second light sensing device, where the samples of thesecond sequence are samples of intensity of the second beam as afunction of time, where the second sequence of samples representsvariation of background light level.

Various additional embodiments are described in the following numberedparagraphs.

3.1 A system for operating on light, the system comprising:

a first optical subsystem configured to receive the light and toseparate the received light into a first light stream and a second lightstream;

a light modulation unit configured to modulate the first light streamwith a series of spatial patterns in order to produce a modulated lightstream, wherein the light modulation unit includes a plurality of lightmodulating elements configured to modulate corresponding portions of thefirst light stream, wherein each of the spatial patterns specifies anamount of modulation for each of the light modulating elements;

a first light sensing device configured to receive the modulated lightstream and to generate a first sequence of samples of intensity of themodulated light stream as a function of time;

a second light sensing device configured to generate a second sequenceof samples of intensity of the second light stream as a function oftime, wherein the second sequence of samples corresponds to variation inbackground light level.

3.2. The system of paragraph 3.1, further comprising a memory thatstores the first sequence and the second sequence of samples, whereinthe first sequence and second sequence of samples are usable toconstruct an image or a video sequence.

3.3. The system of paragraph 3.1, wherein the samples of the secondsequence are used to compensate the samples of the first sequence forthe variation in the background light level, wherein the compensatedsamples are usable to construct the image or a video sequence.

3.4 The system of paragraph 3.1, further comprising a processing unitconfigured to compensate the samples of the first sequence for thevariation in the background light level using the samples of the secondsequence.

3.5. The system of paragraph 3.4, wherein the processing unit is furtherconfigured to generate an image or a video sequence using thecompensated samples.

3.6 The system of paragraph 3.4, wherein the processing unit isconfigured to transmit the compensated samples to a receiver through acommunication channel, wherein the receiver is configured to constructan image or a video sequence based on the compensated samples.

3.7 The system of paragraph 3.1, wherein the plurality of lightmodulating elements are mirrors whose orientations are independentlycontrollable.

3.8 The system of paragraph 3.1, wherein the plurality of lightmodulating elements are elements whose transmittances are independentlycontrollable.

3.9 The system of paragraph 3.1, further comprising a second opticalsubsystem configured to direct the modulated light stream onto a lightsensing surface of the first light sensing device.

3.10 The system of paragraph 3.1, further comprising a third opticalsubsystem configured to receive the second light stream, and to directthe second light stream onto a light sensing surface of the second lightsensing device.

3.11 The system of paragraph 3.1, wherein the plurality of mirrors arearranged in an two-dimensional array.

3.12 The system of paragraph 3.1, wherein the spatial patterns of theseries are pseudo-random patterns.

3.13 The system of paragraph 3.1, wherein the first light sensing deviceincludes one or more light sensing elements.

3.14 The system of paragraph 3.1, wherein the second light sensingdevice includes one or more light sensing elements.

3.15 A non-transitory computer-readable memory medium storing programinstructions, wherein the program instructions, when executed by acomputer system, cause the computer system to: supply a series ofspatial patterns to a light modulation unit, wherein the lightmodulation unit is configured to modulate a first light stream with theseries of spatial patterns to obtain a modulated light stream, whereinthe first light stream is produced by an optical subsystem thatseparates an input light stream into the first light stream and a secondlight stream; receive a first sequence of samples from a first lightsensing device, wherein the first sequence of samples representsintensity of the modulated light stream as a function of time; receive asecond sequence of samples from a second light sensing device, whereinthe samples of the second sequence are samples of intensity of thesecond light stream as a function of time, wherein the second sequenceof samples represents variation of background light level; andcompensate the first sequence of samples for the variation of thebackground light level using the second sequence of samples in order toobtain a compensated sequence of samples, wherein the compensatedsamples are usable to construct an image or a video sequence.

3.16 A system for operating on light, the system comprising:

a first optical subsystem configured to receive the light and toseparate the received light into a first beam and a second beam;

a plurality of mirrors configured to receive corresponding portions ofthe first beam, wherein each of the mirrors is configured tocontrollably switch between two orientation states, wherein each of themirrors is configured to: (a) reflect the corresponding portion of thefirst beam onto a sensing path when the mirror is in a first of the twoorientation states and (b) reflect the corresponding portion of thefirst beam away from the sensing path when the mirror is in a second ofthe two orientation states;

a control unit configured to drive the orientation states of the mirrorsthrough a series of patterns, wherein each of the patterns of the seriesspecifies an orientation state for each of the mirrors;

a first light sensing device configured to receive light portionsreflected onto the sensing path by mirrors in the first orientationstate and to generate a first sequence of samples representing acumulative intensity of the light portions reflected onto the sensingpath as a function of time;

a second light sensing device configured to generate a second sequenceof samples of intensity of the second beam as a function of time,wherein the second sequence of samples corresponds to variation inbackground light level.

3.17. The system of paragraph 3.16, further comprising a second opticalsubsystem configured to receive the light portions reflected onto thesensing path and to direct the received light portions onto a lightsensing surface of the first light sensing device.

3.18. The system of paragraph 3.16, further comprising a third opticalsubsystem configured to receive the second beam, and to direct thesecond beam onto a light sensing surface of the second light sensingdevice.

Low-Pass Filtering the Compressive Sensing Measurements

In one set of embodiments, a system 1800 for operating on incident lightL from an environment may be configured as shown in FIG. 18A.

System 1800 includes the light modulation unit 110, the light sensingdevice 130 and the processing unit 150 as described above in connectionwith system 100. (Any subset of the features, embodiments and elementsdiscussed above with respect to system 100 and system realization 200may be incorporated into system 1800.)

As variously described above, the light modulation unit 110 isconfigured to modulate the light stream L with a series of spatialpatterns in order to produce a modulated light stream MLS. The spatialpatterns may be as variously described above. The light sensing device130 is configured to receive the modulated light stream and to generatea sequence of samples {I_(MLS)(k)} representing intensity of themodulated light stream as a function of time.

In system 1800, the processing unit 150 is configured to low-pass filterthe sequence of samples {I_(MLS)(k)} to obtain a filtered sequence ofsamples {I_(F)(k)}. The filtered sequence of samples {I_(F)(k)}represents variation in background light level (or background lightlevel as a function of time). The filtering may be designed to remove(or decrease or attenuate) high-frequency fluctuations that are presentin the sequence {I_(MLS)(k)}, e.g., high-frequency fluctuations due tothe modulation of the light stream L by the series of spatial patterns.(Even when the incident light stream is constant over space and time,there will be high-frequency fluctuations in the sample sequence{I_(MLS)(k)} due to the modulation by the spatial patterns.)

The processing unit 150 may perform the low-pass filtering in any of awide variety of ways that are known in the art of signal processing,e.g., using any of a wide variety of FIR filters or IIR filters ornon-linear filters. The low-pass filtering may be performed in software(on a microprocessor) or in custom-designed analog or digital circuitryor in a programmable hardware element such as an FPGA.

In some embodiments, the processing unit 150 may periodically (orintermittently or continuously) perform spectrum analysis on the samples{I_(MLS)(k)} and dynamically determine a filter (or parameters of afilter) to be applied to the samples {I_(MLS)(k)}.

In one alternative embodiment, the low-pass filtering is performed inthe analog domain on the analog signal I_(MLS)(t) generated by the lightsensing device 130 (before the signal is digitized by the A/Dconverter), in order to obtain a filtered analog signal which can thenbe digitized to create the filtered sample sequence {I_(F)(k)}.

The sequence {I_(MLS)(k)} and the filtered sequence {I_(F)(k)} may bestored in a memory, e.g., a memory resident in the system 1800 orelsewhere. The sequence {I_(MLS)(k)} and the filtered sequence{I_(F)(k)} are usable to construct an image or a video sequence. Inparticular, the filtered sequence {I_(F)(k)} may be used to compensatethe sequence {I_(MLS)(k)} for the variation in background light level,e.g., as variously described above. The compensated samples{I_(CMPN)(k)} may then be used to construct the image or the videosequence, e.g., based on any of the methods known in the field ofcompressive sensing.

In some embodiments, processing unit 150 may be configured to performthe action of compensating the samples {I_(MLS)(k)} for the variation inthe background light level using the samples of the filtered sequence{I_(F)(k)}. The compensated samples {I_(CMPN)(k)} may be stored in amemory, e.g., a memory resident in system 1800 or elsewhere.

In some embodiments, any variations in the background light level may beinferred by computing local mean values of the samples {I_(MLS)(k)}(e.g., via a sliding window or other approach). Then samples{I_(MLS)(k)} may be correcting for variations in the local mean value incomparison to a global mean of the samples {I_(MLS)(k)}. The processingunit 150 may correct for the light level variations by re-centering allof the measurements to the same global mean value. The local means maybe implemented by low-pass filtering so that variations on the order ofthe pattern-application frequency are filtered out, leaving only slowervariations that correspond to variations in the local mean value.

In some embodiments, the compensated samples {I_(CMPN)(k)} aretransmitted through a communication channel to receiver. (The receivermay be configured to construct an image or a video sequence based on thecompensated samples.) In one embodiment, the processing unit 150 is alsoconfigured to construct the image or the video sequence based on thecompensated samples, i.e., in addition to being configured to transmitthe compensated samples.

In some embodiments, system 1800 (e.g., processing unit 150) may beconfigured to transmit the sequence of samples {I_(MLS)(k)} to areceiver through a communication channel. The receiver may be configuredto perform the action of low-pass filtering the samples {I_(MLS)(k)} toobtain the filtered sequence {I_(F)(k)} that represents variation ofbackground light level.

As shown in FIG. 18B, system 1800 may include the control unit 120 asdescribed above in connection with system 100. The control unit isconfigured to supply the series of spatial patterns to the lightmodulation unit 110.

FIG. 18C shows one embodiment of system 1800, where the light modulationunit 110 is realized by the mirrors 110M as variously described above.Each of the mirrors 110M is configured to controllably switch betweentwo orientation states, and to (a) reflect a corresponding portion ofthe light L onto the sensing path 115 when the mirror is in a first ofthe two orientation states and (b) reflect the corresponding portion ofthe light away from the sensing path when the mirror is in a second ofthe two orientation states. Control unit 120 is configured to drive theorientation states of the mirrors through a series of patterns, whereeach of the patterns of the series specifies an orientation state foreach of the mirrors.

In one set of embodiments, a method 1900 for operating on a stream oflight may include the actions shown in FIG. 19A.

Action 1910 includes modulating a received light stream with a series ofspatial patterns in order to produce a modulated light stream, e.g., asvariously described above.

Action 1915 includes generating a first sequence of samples {I_(MLS)(k)}representing intensity of the modulated light stream as a function oftime, e.g., as variously described above.

Action 1920 includes applying a low-pass filter to the first sequence ofsamples to obtain a filtered sequence of samples {I_(F)(k)}, e.g., asvariously described above. The filtered sequence of samples representsvariation in background light level. In various embodiments, thelow-pass filter may be an FIR filter, an IIR filter, or a non-linearfilter.

In some embodiments, method 1900 also includes storing the sequence{I_(MLS)(k)} and the filtered sequence {I_(F)(k)} in a memory. Thesequence {I_(MLS)(k)} and the filtered sequence {I_(F)(k)} are usable toconstruct an image or a video sequence.

In some embodiments, method 1900 also includes compensating the sequenceof samples {I_(MLS)(k)} for the variation of the background light levelusing the filtered sequence of samples {I_(F)(k)}.

In some embodiments, method 1900 also includes constructing the image orthe video sequence based on the compensated samples.

In some embodiments, method 1900 also includes transmitting thecompensated samples to a receiver through a communication channel. (Thereceiver may be configured to construct the image or the video sequencebased on the compensated samples, and/or, to perform an inference taskas variously described above.)

In some embodiments, a non-transitory computer-readable memory mediummay be configured so that it stores program instructions, where theprogram instructions, when executed by a computer system, cause thecomputer system to: (1) supply a series of spatial patterns to a lightmodulation unit (e.g., the light modulation unit 110 as variouslydescribed above), where the light modulation unit is configured tomodulate a received light stream with the series of spatial patterns toobtain a modulated light stream; (2) receive a first sequence of samplesfrom a light sensing device (e.g., the light sensing device 130 asvariously described above), where the first sequence of samplesrepresents intensity of the modulated light stream as a function oftime; and (3) apply a low-pass filter to the first sequence of samplesto obtain a filtered sequence of samples, where the filtered sequence ofsamples represents variation in background light level.

In one set of embodiments, a method 1960 (e.g., a method for operatingon light) may include the actions shown in FIG. 19B.

Action 1960 includes controlling a plurality of mirrors, where themirrors are configured to reflect corresponding portions of a receivedbeam of light, where each of the mirrors is configured to controllablyswitch between two orientation states, e.g., as variously describedabove. The action of controlling the mirrors includes driving themirrors through a series of spatial patterns, where each of the spatialpatterns specifies an orientation state for each of the mirrors.

Action 1965 includes receiving a sequence of samples {I_(MLS)(k)} from alight sensing device, e.g., as variously described above. The sequenceof samples {I_(MLS)(k)} represents cumulative intensity of lightportions reflected by mirrors in a first of the two orientation statesas a function of time.

Action 1970 includes filtering the sequence of samples {I_(MLS)(k)} witha low-pass filter to obtain a filtered sequence of samples {I_(F)(k)},e.g., as described above. The filtered sequence of samples representsvariation in background light level.

Various additional embodiments are described in the following numberedparagraphs.

4.1 A system for operating on a stream of light, the system comprising:

a light modulation unit configured to modulate the light stream with aseries of spatial patterns in order to produce a modulated light stream,wherein the light modulation unit includes a plurality of lightmodulating elements configured to modulate corresponding portions of thelight stream, wherein each of the spatial patterns specifies an amountof modulation for each of the light modulating elements;

a light sensing device configured to receive the modulated light streamand to generate a first sequence of samples representing intensity ofthe modulated light stream as a function of time; and

a processing unit configured to low-pass filter the first sequence ofsamples to obtain a filtered sequence of samples, wherein the filteredsequence of samples represents variation in background light level.

4.2. The system of paragraph 4.1, further comprising a memory thatstores the first sequence of samples and the filtered sequence ofsamples, wherein the first sequence and the filter sequence of samplesare usable to construct an image or a video sequence.

4.3. The system of paragraph 4.2, wherein the samples of the filteredsequence are used to compensate the samples of the first sequence forthe variation in background light level, wherein the compensated samplesare useable to construct the image or the video sequence.

4.4. The system of paragraph 4.1, wherein the processing unit isconfigured to compensate the samples of the first sequence for thevariation in the background light level using the samples of thefiltered sequence.

4.5. The system of paragraph 4.4, wherein the processing unit isconfigured to construct an image or a video sequence based on thecompensated samples.

4.6 The system of paragraph 4.4, wherein the processing unit isconfigured to transmit the compensated samples to a receiver through acommunication channel, wherein the receiver is configured to constructan image or a video sequence based on the compensated samples.

4.7 The system of paragraph 4.1, wherein the plurality of lightmodulating elements are mirrors whose orientations are independentlycontrollable.

4.8 The system of paragraph 4.1, wherein the plurality of lightmodulating elements are elements whose transmittances are independentlycontrollable.

4.9 The system of paragraph 4.1, further comprising an optical subsystemconfigured to direct the modulated light stream onto a light sensingsurface of the light sensing device.

4.10 The system of paragraph 4.1, wherein the spatial patterns of theseries are pseudo-random patterns.

4.11 The system of paragraph 4.1, wherein the plurality of lightmodulating elements are arranged in a two-dimensional array.

4.12 The system of paragraph 4.1, wherein the light sensing deviceincludes one or more light sensing elements.

4.13 A non-transitory computer-readable memory medium storing programinstructions, wherein the program instructions, when executed by acomputer system, cause the computer system to: supply a series ofspatial patterns to a light modulation unit, wherein the lightmodulation unit is configured to modulate a received light stream withthe series of spatial patterns to obtain a modulated light stream;receive a first sequence of samples from a light sensing device, whereinthe first sequence of samples represents intensity of the modulatedlight stream as a function of time; and apply a low-pass filter to thefirst sequence of samples to obtain a filtered sequence of samples,wherein the filtered sequence of samples represents variation inbackground light level.

4.14 A system for operating on light, the system comprising:

a plurality of mirrors, wherein each of the mirrors is configured tocontrollably switch between two orientation states, wherein each of themirrors is configured to (a) reflect a corresponding portion of thelight onto a sensing path when the mirror is in a first of the twoorientation states and (b) reflect the corresponding portion of thelight away from the sensing path when the mirror is in a second of thetwo orientation states;

a control unit configured to drive the orientation states of the mirrorsthrough a series of patterns, wherein each of the patterns of the seriesspecifies an orientation state for each of the mirrors;

a light sensing device configured to receive light portions reflectedonto the sensing path by mirrors in the first orientation state and togenerate a first sequence of samples representing a cumulative intensityof the light portions reflected onto the sensing path as a function oftime;

a processing unit configured to low-pass filter the first sequence ofsamples to obtain a filtered sequence of samples, wherein the filteredsequence of samples represents variation in background light level.

4.15 The system of paragraph 4.14, further comprising an opticalsubsystem configured to receive the light portions reflected onto thesensing path, and to direct the received light portions onto a lightsensing surface of the light sensing device.

High-Pass Filtering with Envelope Detection to Estimate Background LightSignal

In some situations, the sequence of samples {I_(MLS)(k)} captured by thelight sensing device 130 may be modeled as the product of a backgroundlight signal B(k) and an information-bearing signal V(k):I _(MLS)(k)=B(k)*V(k).The information-bearing signal V(k) may include a DC component V_(DC)and an dynamic component V_(AC)(k):V(k)=V _(DC) +V _(AC)(k),which implies thatI _(MLS)(k)=B(k)*(V _(DC) +V _(AC)(k)).

Furthermore, the energy of the background light signal B(k) may beconcentrated mostly over low frequencies, and the energy of the dynamiccomponent V_(AC)(k) may be concentrated mostly over relatively higherfrequencies. Thus, the application of a high-pass filter to the sequenceI_(MLS)(k) yields as estimate of the product B(k)*V_(AC)(k):HP(k)=HighPassFilter{I _(MLS)(k)}≈B(k)*V _(AC)(k),provided the cutoff frequency of the high-pass filter is appropriatelychosen, i.e., chosen to be above the frequency band where most of thespectral mass of B(k) is concentrated and below the frequency band wheremost of the spectral mass of the V_(AC)(k) is concentrated.

The background light signal B(k) may then be estimated by performingenvelope detection on the highpass filtered signal HP(k):B _(est)(k)=EnvelopeDetect{HP(k)}≈B(k).

In some embodiments, system 1800 (e.g., processing unit 150) may beconfigured to perform the high-pass filtering and envelope detection inorder to obtain the estimated background light signal B_(est)(k), e.g.,as an alternative to the low-pass filtering described above inconnection with FIG. 18A. These embodiments may incorporate any subsetof the features, embodiments and elements described above with respectto system 100 and system realization 200.

The estimated background light signal B_(est)(k) may be used tocompensate the samples of the sequence {I_(MLS)(k)} for variation in thebackground light level, e.g., as variously described above. (Processingunit 150 may be configured to perform that compensation operation.) Thecompensated samples {I_(CMPN)(k)} may be used to construct an image orvideo sequence, and/or, used to perform an inference task as variouslydescribed above. The image/video construction operation and/or theinference task(s) may be performed by system 1800 (e.g., by processingunit 150) and/or by a remote system. System 1800 may include atransmitter to transmit the compensated samples to the remote systemthrough a communication channel.

In some embodiments, the high-pass filtering is performed in the analogdomain, i.e., on the analog signal I_(MLS)(t) produced by the lightsensing device 130 prior to digitization. Furthermore, in someembodiments, the envelope detection is also performed in the analogdomain, i.e., on the output of the analog high-pass filter.

In some embodiments, the high-pass filtering operation may beprogrammable, so that different high-pass filters may be employed indifferent contexts, e.g., for different rates of pattern modulation(i.e., different rates of applying the spatial patterns to the lightmodulation unit), or for different A/D sampling rates, or for differentnoise environments, or for different sets of spatial patterns, or fordifferent spectral bands. In some embodiments, the envelope detectionmay also be programmable.

The high-pass filter may have any of various forms known in the art ofsignal processing.

The envelope detection may be performed by any of various methods knownin the art of signal processing.

In some embodiments, a method 1972 for operating on a stream of lightmay involve the actions shown in FIG. 19C.

Action 1974 may include modulating the light stream with a series ofspatial patterns in order to produce a modulated light stream, e.g., asvariously described above.

Action 1975 may include generating a first signal representing intensityof the modulated light stream as a function of time, e.g., as variouslydescribed above.

Action 1976 may include applying a high-pass filter to the first signalto obtain a filtered signal, e.g., as variously described above.

Action 1977 may include performing envelope detection on the filteredsignal to obtain a detected signal, e.g., as variously described above.The detected signal represents variation in background light level. (Indifferent embodiments, the first signal, the filtered signal and thedetected signal may be analog signals, discrete-time signals or acombination of analog signals and discrete-time signals.) The detectedsignal may be used to compensate the first signal for the variation inbackground light level, e.g., as variously described above, resulting ina compensated signal. Samples of the compensated signal comprise anencoded (e.g., compressed) representation of an image or video sequence.The samples of the compensated signal are usable to construct the imageor the video sequence, and/or, to perform one or more inference tasks asvariously described above.

In one embodiment of method 1972, the high-pass filter is an FIR filter(or alternatively, an IIR filter).

In one embodiment, method 1972 may also include storing samples of thefirst signal and samples of the detected signal in a memory.

In one embodiment, method 1972 includes compensating samples of thefirst signal for the variation in the background light level using thesamples of the detected signal, e.g., as variously described above.

In one embodiment, method 1972 also includes constructing an image or avideo sequence based on the compensated samples, e.g., as variouslydescribed above.

In one embodiment, method 1972 also includes transmitting thecompensated samples to a receiver through a communication channel, e.g.,as variously described above.

In some embodiments, a method 1980 for operating on a stream of lightmay involve the actions shown in FIG. 19D.

Action 1982 may include modulating the light stream with a series ofspatial patterns in order to produce a modulated light stream, e.g., asvariously described above.

Action 1983 may include generating a first sequence of samplesrepresenting intensity of the modulated light stream as a function oftime, e.g., as variously described above.

Action 1984 may include applying a high-pass filter to the firstsequence of samples to obtain a filtered sequence of samples, e.g., asvariously described above.

Action 1985 may include performing envelope detection on the filteredsequence of samples to obtain a detected sequence of samples. Thedetected sequence of samples represents variation in background lightlevel. The detected sequence of samples may be used to compensate thefirst sequence of samples for the variation in background light level(e.g., as variously described above), resulting in a compensatedsequence of samples. The compensated sequence of samples comprises anencoded (e.g., compressed) representation of an image or video sequence.The compensated sequence is usable to construct the image or the videosequence, and/or, to perform one or more inference tasks as variouslydescribed above.

Various additional embodiments are described in the following numberedparagraphs.

4B.1 A system for operating on a stream of light, the system comprising:

a light modulation unit configured to modulate the light stream with aseries of spatial patterns in order to produce a modulated light stream,wherein the light modulation unit includes a plurality of lightmodulating elements configured to modulate corresponding portions of thelight stream, wherein each of the spatial patterns specifies an amountof modulation for each of the light modulating elements;

a light sensing device configured to receive the modulated light streamand to generate a first signal representing intensity of the modulatedlight stream as a function of time; and

a processing unit configured to high-pass filter the first signal toobtain a filtered signal, wherein the processing unit is also configuredto perform envelope detection on the filtered signal to obtain adetected signal, wherein the detected signal represents variation inbackground light level.

4B.2. The system of paragraph 4B.1, further comprising a memory thatstores samples of the first signal and samples of the detected signal,wherein the samples of the first signal and the samples of the detectedsignal are usable to construct an image or a video sequence.

4B.3. The system of paragraph 4B.2, wherein the samples of the detectedsignal are used to compensate the samples of the first signal for thevariation in background light level, wherein the compensated samples areuseable to construct the image or the video sequence.

4B.4. The system of paragraph 4B.1, wherein the processing unit isconfigured to compensate samples of the first signal for the variationin the background light level using samples of the detected signal.

4B.5. The system of paragraph 4B.4, wherein the processing unit isconfigured to construct an image or a video sequence based on thecompensated samples.

4B.6 The system of paragraph 4B.4, wherein the processing unit isconfigured to transmit the compensated samples to a receiver through acommunication channel, wherein the receiver is configured to constructan image or a video sequence based on the compensated samples.

4B.7 The system of paragraph 4B.1, further comprising an opticalsubsystem configured to direct the modulated light stream onto a lightsensing surface of the light sensing device.

4B.8 The system of paragraph 4B.1, wherein the plurality of lightmodulating elements are arranged in a two-dimensional array.

4B.9 The system of paragraph 4B.1, wherein the light sensing deviceincludes one or more light sensing elements.

4B.10 A non-transitory computer-readable memory medium storing programinstructions, wherein the program instructions, when executed by acomputer system, cause the computer system to: supply a series ofspatial patterns to a light modulation unit, wherein the lightmodulation unit is configured to modulate a received light stream withthe series of spatial patterns to obtain a modulated light stream;receive a first signal from a light sensing device, wherein the firstsignal represents intensity of the modulated light stream as a functionof time; apply a high-pass filter to the first signal to obtain afiltered signal; and perform envelope detection on the filtered signalto obtain a detected signal, wherein the filtered signal representsvariation in background light level.

High-Pass Filtering to Estimate Compressive Sensing Information andAttenuate Contribution Due to Background Light Variation

In some situations, the sequence of samples {I_(MLS)(k)} captured by thelight sensing device 130 may be modeled as the sum of aninformation-bearing signal r(k) and a disturbance signal b(k). Thedisturbance signal may be due at least in part to variation in thebackground light level (ambient illumination) in the scene beingacquired. The disturbance signal may also be due in part tolow-frequency electrical noise, e.g., AC power noise. The energy of thedisturbance signal may be concentrated mostly over low frequencies, andthe energy of the information-bearing signal may be concentrated mostlyover relatively higher frequencies. Consequently, theinformation-bearing signal r(k) may be estimated by applying a high-passfilter to the sample sequence I_(MLS)(k). The cutoff frequency of thehigh-pass filter is selected so that the high-pass filter substantiallyeliminates the disturbance signal and substantially passes theinformation-bearing signal r(k). (For example, the cutoff frequency ofthe high-pass filter may be chosen to be above the frequency band wheremost of the spectral mass of b(k) is concentrated and below thefrequency band where most of the spectral mass of the r(k) isconcentrated.)

In some embodiments, a system 1988 may be configured to include thelight modulation unit 110, the light sensing device 130 and theprocessing unit 150, as shown in FIG. 19E. The system 1988 (e.g., theprocessing unit 150) may be configured to perform the above-describedhigh-pass filtering in order to estimate the information-bearing signalr(k). (System 1988 may incorporate any subset of the features,embodiments and elements described above with respect to system 100 andsystem realization 200.) The samples of the filtered signal (i.e., theoutput of the high-pass filter) may be processed to construct an image(or a video sequence), and/or, to perform one or more inference tasks asvariously described above. The image/video construction operation and/orthe inference task(s) may be performed by the system 1988 (e.g., byprocessing unit 150) and/or by a remote system. The system 1988 mayinclude a transmitter to transmit the samples of the filtered signal tothe remote system through a communication channel.

In some embodiments, the high-pass filtering operation is performed inthe analog domain, i.e., on the analog signal I_(MLS)(t) produced by thelight sensing device 130 prior to digitization.

In some embodiments, the high-pass filtering operation may beprogrammable, so that different high-pass filters may be employed indifferent contexts, e.g., for different rates of pattern modulation(i.e., different rates of applying the spatial patterns to the lightmodulation unit), or for different A/D sampling rates, or for differentnoise environments, or for different sets of spatial patterns, or fordifferent spectral bands.

The high-pass filter may have any of various forms known in the art ofsignal processing. For example, in various embodiments, the high-passfilter may be an FIR filter, an IIR filter, or some sort of non-linearfilter.

In some embodiments, the processing unit 150 may periodically (orintermittently or continuously) perform spectrum analysis on the samples{I_(MLS)(k)} and dynamically determine a high-pass filter (or parametersof a high-pass filter) to be applied to the samples {I_(MLS)(k)}.

The system 1988 may include memory for storing the samples of thefiltered signal.

In one alternative embodiment, system 1988 (e.g., processing unit 150)may be configured to estimate the disturbance signal b(k) by low-passfiltering the sequence of samples I_(MLS)(k). The estimated disturbancesignal b_(est)(k) (i.e., the output of the low-pass filtering operation)may then be subtracted from the sequence of samples I_(MLS)(k) to obtainan estimate r_(est)(k) for the information-bearing signal r(k), e.g.,according to the relation:r _(est)(k)=I _(MLS)(k)−b _(est)(k).

In some embodiments, a method 1990 for operating on a stream of lightmay involve the actions shown in FIG. 19F. Method 1990 may performed,e.g., using system 1988.

Action 1991 may include modulating the light stream with a series ofspatial patterns in order to produce a modulated light stream, e.g., asvariously described above.

Action 1992 may include generating a first signal representing intensityof the modulated light stream as a function of time, e.g., as variouslydescribed above.

Action 1993 may include applying a high-pass filter to the first signalto obtain a filtered signal, where the action of applying the high-passfilter attenuates (or decreases or removes) an effect of backgroundlight variation from the first signal. The samples of the filteredsignal comprise an encoded (e.g., compressed) representation of an imageor video sequence. The samples of the filtered signal are usable toconstruct the image or the video sequence, and/or, to perform one ormore inference tasks, as variously described above. The signals may bediscrete-time signals, or analog signals, or a combination of analogsignals and discrete-time signals.

In one embodiment, the method 1990 may also include storing the samplesof the filtered signal in a memory.

In one embodiment, the method 1990 may also include constructing theimage or the video sequence based on the samples of the filtered signal.

In one embodiment, the method 1990 may also include transmitting thesamples of the filtered signal to a receiver through a communicationchannel.

In some embodiments, a method for operating on a stream of light mayinvolve: (a) modulating the light stream with a series of spatialpatterns in order to produce a modulated light stream; (b) generating afirst sequence of samples representing intensity of the modulated lightstream as a function of time; and (c) applying a high-pass filter to thefirst sequence of samples to obtain a filtered sequence of samples. Thefiltered sequence of samples is usable to construct an image or a videosequence, and/or, to perform one or more inference tasks, as variouslydescribed above.

Various additional embodiments are described in the following numberedparagraphs.

5.1 A system for operating on a stream of light, the system comprising:

a light modulation unit configured to modulate the light stream with aseries of spatial patterns in order to produce a modulated light stream,wherein the light modulation unit includes a plurality of lightmodulating elements configured to modulate corresponding portions of thelight stream, wherein each of the spatial patterns specifies an amountof modulation for each of the light modulating elements;a light sensing device configured to receive the modulated light streamand to generate a first signal representing intensity of the modulatedlight stream as a function of time; anda processing unit configured to high-pass filter the first signal toobtain a filtered signal, wherein the high-pass filtering attenuates aneffect of background light variation from the first signal, whereinsamples of the filtered signal comprise an encoded representation of animage or video sequence.5.2. The system of paragraph 5.1, further comprising a memory thatstores the samples of the filtered signal, wherein the samples of thefilter signal are usable to construct the image or the video sequence.5.3. The system of paragraph 5.1, wherein the processing unit isconfigured to construct the image or the video sequence based on thesamples of the filtered signal.5.4 The system of paragraph 5.1, wherein the processing unit isconfigured to transmit the samples of the filtered signal to a receiverthrough a communication channel.5.5 The system of paragraph 5.1, further comprising an optical subsystemconfigured to direct the modulated light stream onto a light sensingsurface of the light sensing device.5.6 The system of paragraph 5.1, wherein the light sensing deviceincludes one or more light sensing elements.5.7 A non-transitory computer-readable memory medium storing programinstructions, wherein the program instructions, when executed by acomputer system, cause the computer system to: supply a series ofspatial patterns to a light modulation unit, wherein the lightmodulation unit is configured to modulate a received light stream withthe series of spatial patterns to obtain a modulated light stream;receive a first signal from a light sensing device, wherein the firstsignal represents intensity of the modulated light stream as a functionof time; and apply a high-pass filter to the first signal to obtain afiltered signal, wherein samples of the filtered signal comprise anencoded representation of an image or video sequence.Light Power Meter to Measure Background Light Level Variation

In one set of embodiments, a system 2000 may be configured as shown inFIG. 20A. System 2000 includes the light modulation unit 110 and thelight sensing device 130 as described above in connection with system100, and also includes a light power meter 2010. (Any subset of thefeatures, embodiments and elements discussed above with respect tosystem 100 and system realization 200 may be incorporated into system2000.)

The light modulation unit 110 is configured to receive a first lightstream L₁ from the environment through a first optical input path 1305and to modulate the first light stream with a series of spatial patternsin order to produce a modulated light stream MLS, e.g., as variouslydescribed above. The light sensing device 130 is configured to receivethe modulated light stream and to generate a first sequence of samples{I_(MLS)(k)} representing intensity of the modulated light stream as afunction of time.

The light power meter 2010 is configured to receive a second lightstream L₂ from the environment via a second optical input path 1307 thatis separate from the first optical input path 1305, and to capture asequence of measurements {I_(BGL)(k)} of power (or intensity) of thesecond light stream over time. The sequence of power measurements{I_(BGL)(k)} represents variation of background light level in theenvironment.

The optical input path 1305 and the optical input path 1307 may beisolated from each other optically so that photons that have entered theoptical input path 1305 from the environment will have limited abilityto be detected at light power meter 2010, and so that photons that haveentered the optical input path 1307 from the environment will havelimited ability to be detected at the light sensing device 130.

In some embodiments, the optical input path 1305 and the optical inputpath 1307 may be configured so that the first light stream L₁ and thesecond light stream L₂ are received from same general direction. Inother words, the optical input path 1307 may have a field of view intothe environment that points in the same general direction as the fieldof view of the optical input path 1305. Thus, the sequence of powermeasurements {I_(BGL)(k)} may be representative of background lightlevel variation for approximately the same portion of the environmentthat is being acquired by the optical input path 1305.

The light power meter 2010 may be implemented in various ways. In someembodiments, the light power meter may be configured the same as orsimilar to light sensing device 130. For example, the light power metermay include the same (or similar) light sensing hardware as lightsensing device 130. In another embodiment, the light power meter 2010includes a light sensing device that is configured differently fromlight sensing device 130. For example, light power meter 2010 may uselight-sensing hardware that is of a lower-cost and/or lower performanceand/or lower-complexity than light sensing device 130.

The light power meter 2010 is preferably calibrated so that its readingsare meaningfully related to light levels in the first beam.

In one embodiment, the optical input path 1307 by which light L₂ entersthe system may be of the same design as the optical input path 1305 thatallows light L₁ to enter the system. In another embodiment, the opticalinput path 1307 may be of lower cost and/or lower performance than theoptical path 1305.

The sequence {I_(MLS)(k)} and the sequence of the power measurements{I_(BGL)(k)} may be stored in a memory, e.g., a memory resident in thesystem 2000 or elsewhere. The sequence {I_(MLS)(k)} and the sequence ofpower measurements {I_(BGL)(k)} are usable to construct an image or avideo sequence, and/or, to perform an inference task, as variouslydescribed above. In particular, the sequence of power measurements{I_(BGL)(k)} may be used to compensate the sequence {I_(MLS)(k)} for thevariation in background light level, e.g., as variously described above.The compensated samples {I_(CMPN)(k)} may then be used to construct theimage or the video sequence (e.g., based on any of the methods known inthe field of compressive sensing), and/or, to perform the inferencetask.

As shown in FIG. 20B, system 2000 may also include the control unit 120described above in connection with system 100. The control unit isconfigured to supply the series of spatial patterns to the lightmodulation unit 110.

As shown in FIG. 20C, the system 2000 may also include the processingunit 150 discussed above in connection with system 100. The processingunit 150 may be configured to compensate the samples of the sequence{I_(MLS)(k)} for the variation in background light level using sequenceof power measurements {I_(BGL)(k)}. The compensated samples{I_(CMPN)(k)} are usable to construct an image or a video sequence,e.g., as variously described above. In some embodiments, the processingunit 150 may also be configured to perform the action of constructingthe image or the video sequence based on the compensated samples,and/or, the action of performing an inference task as variouslydescribed above.

FIG. 21 shows one embodiment of system 2000, where the light modulationunit 110 is realized by the set of mirrors 110M as described above inconnection with system 100. The set of mirrors 110M is configured toreceive the light stream L₁ from the environment via an optical inputpath 1305. Each of the mirrors is configured to controllably switchbetween two orientation states, and configured to (a) reflect acorresponding portion of the light stream L₁ onto the sensing path 115when the mirror is in a first of the two orientation states and (b)reflect the corresponding portion of the light stream L₁ away from thesensing path when the mirror is in a second of the two orientationstates. The control unit 120 is configured to drive the orientationstates of the mirrors through the series of spatial patterns, where eachof the patterns of the series specifies an orientation state for each ofthe mirrors. The light sensing device 130 is configured to receive lightportions reflected at any given time onto the sensing path 115 bymirrors in the first orientation state and to generate a sequence ofsamples {I_(MLS)(k)} representing a cumulative intensity of the lightportions reflected onto the sensing path as a function of time. Thesamples {I_(MLS)(k)} may be interpreted as compressive sensingmeasurements.

In one alternative embodiment, system 2000 includes twoseparately-packaged units as shown in FIG. 22. The first unit 2050includes the light modulation unit 110, the light sensing device 130 andprocessing unit 150. The second unit 2060 includes the light power meter2010. The second unit may be configured to couple with the first unitvia a wired or wireless connection. The second unit is configured tocapture the sequence of power measurements {I_(BGL)(k)} (e.g., asvariously described above) and to send the power measurements to thefirst unit or to some other processing agent. The second unit may beconfigured as a small light-weight package that is easy for a user tohold and point. The second unit may be battery powered.

In one set of embodiments, a method 2300 (e.g., a method for obtainingcompressed measurements of a received light stream) may include theactions shown in FIG. 23A.

Action 2310 includes receiving a first light stream L₁ through a firstinput path, e.g., as variously described above.

Action 2315 includes receiving a second light stream L₂ through a secondinput path that is separate from the first optical input path, e.g., asvariously described above.

Action 2320 includes modulating the first light stream L₁ with a seriesof spatial patterns in order to produce a modulated light stream, e.g.,as variously described above.

Action 2325 includes generating (or capturing) a first sequence ofsamples {I_(MLS)(k)} representing intensity of the modulated lightstream as a function of time, e.g., as variously described above.

Action 2330 includes generating (or capturing) a sequence ofmeasurements {I_(BGL)(k)} of power (or intensity) of the second lightstream L₂ over time, e.g., as variously described above. The sequence ofpower measurements {I_(BGL)(k)} represents variation in background lightlevel.

The samples {I_(MLS)(k)} and the measurements {I_(BGL)(k)} may becaptured in parallel. In other words, the light power meter may capturethe measurements {I_(BGL)(k)} while the light sensing device capturesthe samples {I_(MLS)(k)}.

In some embodiments, the samples {I_(MLS)(k)} and the measurements{I_(BGL)(k)} are captured at the same rate, e.g., in response to samesample clock. In other embodiments, the samples {I_(MLS)(k)} and themeasurements {I_(BGL)(k)} are captured at different rates. In oneembodiment, the samples {I_(MLS)(k)} may be captured at a rate that isan integer multiple of the rate at which the measurements {I_(BGL)(k)}are captured.

Action 2335 includes compensating the first sequence of samples{I_(MLS)(k)} for the variation in the background light level using thesequence of power measurements, e.g., as variously described above. Thecompensated samples are usable to construct an image or a videosequence, and/or, to perform an inference task as variously describedabove.

In some embodiments, method 2300 may also include storing thecompensated samples in a memory.

In some embodiments, method 2300 may also include constructing an imageor a video sequence using the compensated samples, e.g., as variouslydescribed above.

In some embodiments, method 2300 may also include transmitting thecompensated samples to a receiver through a communication channel. (Thereceiver may be configured to construct the image based on thecompensated samples.)

In some embodiments, a non-transitory computer-readable memory mediummay be configured so that it stores program instructions, where theprogram instructions, when executed by a computer system, cause thecomputer system to: (1) supply a series of spatial patterns to a lightmodulation unit, where the light modulation unit is configured tomodulate a first stream of light with the series of spatial patterns toobtain a modulated light stream, where the first stream of light isreceived through a first optical input path; (2) receive a firstsequence of samples from a light sensing device, where the firstsequence of samples represents intensity of the modulated light streamas a function of time; (3) receive a sequence of power measurements froma light power meter, where the sequence of power measurements aremeasurements of power of a second stream of light over time, where thesecond light stream is received through a second optical input path thatis separate from the first optical input path, where the sequence ofpower measurements represents variation in background light level; and(4) compensate the first sequence of samples for the variation in thebackground light level using the sequence of power measurements, wherethe compensated samples are usable to construct an image or a videosequence, and/or, to perform an inference task, e.g., as variouslydescribed above.

In one embodiment, the program instructions, when executed by a computersystem, also cause the computer system to generate the series of spatialpatterns by executing a random number generator algorithm using one ormore seeds.

In one set of embodiments, a method 2350, e.g., a method for obtainingcompressed measurements of a received light stream, may include theactions shown in FIG. 23B.

Action 2360 includes receiving light L₁ through a first optical inputpath, e.g., as described above.

Action 2365 includes receiving light L₂ through a second optical inputpath that is separate from the first optical input path, e.g., asdescribed above.

Action 2370 includes controlling a plurality of mirrors, where themirrors are configured to reflect corresponding portions of the lightL₁, where each of the mirrors is configured to controllably switchbetween two orientation states, where the operation of controlling themirrors includes driving the mirrors through a series of spatialpatterns, e.g., as variously described above. Each of the spatialpatterns specifies an orientation state for each of the mirrors.

Action 2375 includes receiving a sequence of samples {I_(MLS)(k)} from alight sensing device, e.g., as variously described above. The sequenceof samples {I_(MLS)(k)} represents cumulative intensity of lightportions reflected by mirrors in a first of the two orientation statesas a function of time.

Action 2380 includes receiving a sequence of measurements {I_(BGL)(k)}of power (or intensity) from a light power meter, e.g., as variouslydescribed above. The sequence of power measurements are measurements ofpower of the light L₂ over time. The sequence of power measurementsrepresents variation in background light level (e.g., in the environmentor that portion of the environment that is being acquired.)

Action 2385 includes compensating the sequence of samples {I_(MLS)(k)}for the variation in the background light level using the sequence ofpower measurements {I_(BGL)(k)}. The resulting compensated samples{I_(CMPN)(k)} are usable to construct an image or video sequence,and/or, to perform an inference task, e.g., as variously describedabove.

In one set of embodiments, a non-transitory computer-readable memorymedium may be configured so that it stores program instructions, wherethe program instructions, when executed by a computer system, cause thecomputer system to: (1) control a plurality of mirrors, where themirrors are configured to reflect corresponding portions of light L₁,where the light L₁ is received through a first optical input path, whereeach of the mirrors is configured to controllably switch between twoorientation states, where the action of controlling the mirrors includesdriving the mirrors through a series of spatial patterns, where each ofthe spatial patterns specifies an orientation state for each of themirrors; (2) receive a sequence of samples {I_(MLS)(k)} from a lightsensing device, where the first sequence of samples representscumulative intensity of light portions reflected by mirrors in a firstof the two orientation states as a function of time; (3) receive asequence of power measurements {I_(BGL)(k)} from a light power meter,where the sequence of power measurements are measurements of power oflight L₂ over time, where the second light is received through a secondoptical input path that is separate from the first optical input path,where the sequence of power measurements represents variation inbackground light level; and (4) compensate the sequence of samples{I_(MLS)(k)} for the variation of the background light level using thesequence of power measurements {I_(BGL)(k)}, where the compensatedsamples are usable to construct an image or video sequence, and/or, toperform an inference task, e.g., as variously described above.

Various additional embodiments are described in the following numberedparagraphs.

6.1 A system comprising:

a light modulation unit configured to receive a first stream of lightfrom an environment through a first optical input path and to modulatethe first light stream with a series of spatial patterns in order toproduce a modulated light stream, wherein the light modulation unitincludes a plurality of light modulating elements configured to modulatecorresponding portions of the first light stream, wherein each of thespatial patterns specifies an amount of modulation for each of the lightmodulating elements;

a light sensing device configured to receive the modulated light streamand to generate a first sequence of samples representing intensity ofthe modulated light stream as a function of time; and

a light power meter configured to receive a second stream of light fromthe environment via a second optical input path that is separate fromthe first optical input path, and to capture a sequence of measurementsof power of the second light stream over time.

6.2. The system of paragraph 6.1, further comprising a memory thatstores the first sequence of samples and the sequence of powermeasurements, wherein the first sequence of samples and the sequence ofpower measurements are usable to construct an image or a video sequence.

6.3. The system of paragraph 6.1, wherein power measurements are used tocompensate the samples of the first sequence for variation in backgroundlight level, wherein the compensated samples are usable to construct animage or a video sequence.

6.4. The system of paragraph 6.1, further comprising a processing unitconfigured to compensate the samples of the first sequence for variationin background light level using power measurements, wherein thecompensated samples are usable to construct an image or a videosequence.

6.5. The system of paragraph 6.4, wherein the processing unit isconfigured to construct the image or the video sequence based on thecompensated samples.

6.6 The system of paragraph 6.4, wherein the processing unit isconfigured to transmit the compensated samples to a receiver through acommunication channel, wherein the receiver is configured to constructthe image or the video sequence based on the compensated samples.

6.7 The system of paragraph 6.1, further comprising an optical subsystemconfigured to receive the modulated light stream and to direct themodulated light stream onto a light sensing surface of the light sensingdevice.

6.8. The system of paragraph 6.1, wherein the plurality of lightmodulating elements are arranged in a two-dimensional array.

6.9 The system of paragraph 6.1, wherein the spatial patterns of theseries are pseudo-random patterns.

6.10. The system of paragraph 6.1, wherein the light sensing deviceincludes one or more photodiodes.

6.11 A system comprising:

a plurality of mirrors configured to receive first light from anenvironment via a first optical input path, wherein each of the mirrorsis configured to controllably switch between two orientation states,wherein each of the mirrors is configured to (a) reflect a correspondingportion of the first light onto a sensing path when the mirror is in afirst of the two orientation states and (b) reflect the correspondingportion of the first light away from the sensing path when the mirror isin a second of the two orientation states;

a control unit configured to drive the orientation states of the mirrorsthrough a series of spatial patterns, wherein each of the spatialpatterns of the series specifies an orientation state for each of themirrors;

a light sensing device configured to receive light portions reflectedonto the sensing path by mirrors in the first orientation state and togenerate a first sequence of samples representing a cumulative intensityof the light portions reflected onto the sensing path as a function oftime;

a light power meter configured to receive second light from theenvironment via a second optical input path that is separate from thefirst optical input path, and to capture a sequence of measurements ofpower of the second light over time.

6.12 The system of paragraph 6.11, further comprising a processing unitconfigured to compensate the first sequence of samples for variation inbackground light level using the sequence of power measurements, whereinthe compensated samples are usable to construct an image or a videosequence.

6.13 The system of paragraph 6.11, further comprising an opticalsubsystem configured to receive the light portions reflected onto thesensing path, and to direct the received light portions onto a lightsensing surface of the light sensing device.

Any of the various embodiments described herein may be combined to formcomposite embodiments. For example, any two or more of the variousmechanisms for generating a signal representing the background lightlevel may be combined. Such a composite embodiment would consequentlyproduce two or more signals representing the background light level(i.e., one signal per mechanism), and perhaps operate on those signalsto generate a more robust and/or accurate estimate of the backgroundlight level. This disclosure discusses, among other things, at least sixmechanisms for generating the background light level signal: (1)inclusion of calibration patterns in the series of spatial patterns; (2)sensing light reflected in both (or each of the) orientation states of asystem of controllable mirrors; (3) separating a portion of the incominglight before it arrives at the light modulation unit; (4) low-passfiltering the sample sequence {I_(MLS)(k)}; (5) high-pass filtering andthen envelope detecting the sample sequence {I_(MLS)(k)}; and (6) usinga light power meter having a separate input path.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

What is claimed is:
 1. A system comprising: a light modulation unitconfigured to receive a first stream of light from an environmentthrough a first optical input path and to modulate the first lightstream with a series of spatial patterns in order to produce a modulatedlight stream, wherein the light modulation unit includes a plurality oflight modulating elements configured to modulate corresponding portionsof the first light stream, wherein each of the spatial patternsspecifies an amount of modulation for each of the light modulatingelements; a light sensing device configured to receive the modulatedlight stream and to generate a first sequence of samples representingintensity of the modulated light stream as a function of time; and alight power meter configured to receive a second stream of light fromthe environment via a second optical input path that is separate fromthe first optical input path, and to capture a sequence of measurementsof power of the second light stream over time; wherein the lightmodulation unit and the light sensing unit are packaged in a firsthousing, wherein the light power meter is packaged in a second housingseparate from the first housing.
 2. The system of claim 1, furthercomprising a memory that stores the first sequence of samples and thesequence of power measurements, wherein the first sequence of samplesand the sequence of power measurements are usable to construct an imageor a video sequence.
 3. The system of claim 1, wherein powermeasurements are used to compensate the samples of the first sequencefor variation in background light level, wherein the compensated samplesare usable to construct an image or a video sequence.
 4. The system ofclaim 1, further comprising a processing unit configured to compensatethe samples of the first sequence for variation in background lightlevel using the power measurements, wherein the compensated samples areusable to construct an image or a video sequence.
 5. The system of claim4, wherein the processing unit is configured to compensate the firstsequence of samples by computing a sequence of differences based on thefirst sequence of samples and the sequence of power measurements.
 6. Thesystem of claim 4, wherein the processing unit is configured tocompensate the samples of the first sequence by computing a sequence ofratios based on the first sequence of samples and the sequence of powermeasurements.
 7. The system of claim 4, wherein the processing unit isconfigured to compensate the samples of the first sequence by increasinga given sample of the first sequence when a corresponding powermeasurement of the sequence of power measurements is small.
 8. Thesystem of claim 4, wherein the processing unit is configured to transmitthe compensated samples to a receiver through a communication channel,wherein the receiver is configured to construct the image or the videosequence based on the compensated samples.
 9. The system of claim 1,wherein the plurality of light modulating elements are arranged in atwo-dimensional array.
 10. The system of claim 1, wherein the lightsensing device includes one or more photodiodes.
 11. The system of claim1, wherein the first optical input path is optical isolated from thesecond optical input path.
 12. A system comprising: a plurality ofmirrors configured to receive first light from an environment via afirst optical input path, wherein each of the mirrors is configured tocontrollably switch between two orientation states, wherein each of themirrors is configured to (a) reflect a corresponding portion of thefirst light onto a sensing path when the mirror is in a first of the twoorientation states and (b) reflect the corresponding portion of thefirst light away from the sensing path when the mirror is in a second ofthe two orientation states; a control unit configured to drive theorientation states of the mirrors through a series of spatial patterns,wherein each of the spatial patterns of the series specifies anorientation state for each of the mirrors; a light sensing deviceconfigured to receive light portions reflected onto the sensing path bymirrors in the first orientation state and to generate a first sequenceof samples representing a cumulative intensity of the light portionsreflected onto the sensing path as a function of time; a light powermeter configured to receive second light from the environment via asecond optical input path that is separate from the first optical inputpath, and to capture a sequence of measurements of power of the secondlight over time; a processing unit configured to compensate the samplesof the first sequence for variation in background light level using thepower measurements, wherein the compensated samples are usable toconstruct an image or a video sequence; wherein the processing unit isconfigured to compensate the first sequence of samples by computing asequence of differences based on the first sequence of samples and thesequence of power measurements.
 13. A method comprising: receiving afirst stream of light from an environment through a first optical inputpath; modulating the first light stream with a series of spatialpatterns in order to produce a modulated light stream, wherein saidmodulating is performed by a light modulation unit having a plurality oflight modulating elements configured to modulate corresponding portionsof the first light stream, wherein each of the spatial patternsspecifies an amount of modulation for each of the light modulatingelements; generating a first sequence of samples representing intensityof the modulated light stream as a function of time, wherein saidgenerating is performed by a light sensing device; and receiving asecond stream of light from the environment via a second optical inputpath that is separate from the first optical input path; and capturing asequence of measurements of power of the second light stream over time,wherein said capturing is performed by a light power meter; compensatingthe samples of the first sequence for variation in background lightlevel using the power measurements, wherein the compensated samples areusable to construct an image or a video sequence; wherein saidcompensating includes computing a sequence of ratios based on the firstsequence of samples and the sequence of power measurements.
 14. Themethod of claim 13, wherein said compensating the samples of the firstsequence includes increasing a given sample of the first sequence when acorresponding power measurement of the sequence of power measurements issmall.